2024-03-21T12:54:00.313 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-21T12:54:00.313 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968 --description orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} --name gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi --verbose -- /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968/orig.config.yaml 2024-03-21T12:54:00.343 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T12:54:00.444 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968 branch: wip-guits-testing-2024-03-21-0818 description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} email: gabrioux@ibm.com first_in_suite: false job_id: '7614968' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-03-21-0818 ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 960fbbe4645c2becc8084669be194c95183c7753 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 960fbbe4645c2becc8084669be194c95183c7753 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-2024-03-21-0818 sha1: 960fbbe4645c2becc8084669be194c95183c7753 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 187 sha1: 960fbbe4645c2becc8084669be194c95183c7753 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-03-21-0818 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 960fbbe4645c2becc8084669be194c95183c7753 targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDW2mtFwLHDI53XM5x30fN+QMr0GjYn8Z9SGaI2yhRbTGeZ/a6yDZeKmK9DTnHfqDfbtY6MycCsg8wvR/boRIBQ= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJVRSq3H3HXMpH8erFtBRpPhkjH2VgB31174t5l1L/huEQKypktSEaC/3a2bR1kYBdKOpizdGSX9sRSbgXBDqE= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-21_12:04:17 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-21T12:54:00.444 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa; will attempt to use it 2024-03-21T12:54:00.445 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa/tasks 2024-03-21T12:54:00.445 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-21T12:54:00.447 INFO:teuthology.task.internal:Checking packages... 2024-03-21T12:54:00.472 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '960fbbe4645c2becc8084669be194c95183c7753' 2024-03-21T12:54:00.472 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-21T12:54:00.472 INFO:teuthology.packaging:ref: None 2024-03-21T12:54:00.472 INFO:teuthology.packaging:tag: None 2024-03-21T12:54:00.472 INFO:teuthology.packaging:branch: wip-guits-testing-2024-03-21-0818 2024-03-21T12:54:00.472 INFO:teuthology.packaging:sha1: 960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:00.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-03-21-0818 2024-03-21T12:54:00.701 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2267.g960fbbe4 2024-03-21T12:54:00.703 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-21T12:54:00.710 INFO:teuthology.task.internal:no buildpackages task found 2024-03-21T12:54:00.710 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-21T12:54:00.717 INFO:teuthology.task.internal:Saving configuration 2024-03-21T12:54:00.728 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-21T12:54:00.734 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T12:54:00.761 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:48:52.449483', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDW2mtFwLHDI53XM5x30fN+QMr0GjYn8Z9SGaI2yhRbTGeZ/a6yDZeKmK9DTnHfqDfbtY6MycCsg8wvR/boRIBQ='} 2024-03-21T12:54:00.784 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:48:52.451928', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJVRSq3H3HXMpH8erFtBRpPhkjH2VgB31174t5l1L/huEQKypktSEaC/3a2bR1kYBdKOpizdGSX9sRSbgXBDqE='} 2024-03-21T12:54:00.785 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-21T12:54:00.791 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-21T12:54:00.791 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-21T12:54:00.791 INFO:teuthology.run_tasks:Running task console_log... 2024-03-21T12:54:00.862 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efe50f0e0d0>, signals=[15]) 2024-03-21T12:54:00.863 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-21T12:54:00.869 INFO:teuthology.task.internal:Opening connections... 2024-03-21T12:54:00.869 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-03-21T12:54:00.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:54:00.963 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-03-21T12:54:00.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:54:01.054 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-21T12:54:01.061 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-03-21T12:54:01.079 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-03-21T12:54:01.079 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-03-21T12:54:01.136 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T12:54:01.137 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T12:54:01.138 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-03-21T12:54:01.178 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-03-21T12:54:01.194 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-03-21T12:54:01.195 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-03-21T12:54:01.250 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-03-21T12:54:01.250 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T12:54:01.251 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T12:54:01.252 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-03-21T12:54:01.278 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-21T12:54:01.287 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-21T12:54:01.293 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-21T12:54:01.293 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-03-21T12:54:01.295 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-03-21T12:54:01.311 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-21T12:54:01.317 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-21T12:54:01.317 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-03-21T12:54:01.352 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-03-21T12:54:01.423 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-21T12:54:01.524 INFO:teuthology.run_tasks:Running task kernel... 2024-03-21T12:54:01.538 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-21T12:54:01.538 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-21T12:54:01.538 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T12:54:01.539 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T12:54:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:54:01.555 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-03-21T12:54:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:54:01.556 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-21T12:54:01.612 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-361.el9.x86_64 2024-03-21T12:54:01.613 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-361.el9.x86_64 2024-03-21T12:54:01.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-03-21T12:54:01.614 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-361.el9.x86_64 2024-03-21T12:54:01.614 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-361.el9.x86_64 2024-03-21T12:54:01.614 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-03-21T12:54:03.683 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-21T12:54:03.742 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-21T12:54:05.308 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-21T12:54:05.384 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-21T12:54:06.190 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-21T12:54:09.958 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 6.9 MB/s | 19 MB 00:02 2024-03-21T12:54:10.175 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-21T12:54:12.801 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-21T12:54:14.092 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.1 MB 00:00 2024-03-21T12:54:15.534 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-03-21T12:54:19.354 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 21 MB 00:03 2024-03-21T12:54:24.059 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 806 kB/s | 6.1 MB 00:07 2024-03-21T12:54:24.327 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-21T12:54:25.676 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-21T12:54:26.442 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T12:54:26.442 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T12:54:26.540 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T12:54:26.545 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T12:54:26.546 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-03-21T12:54:26.547 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T12:54:26.547 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 76 M 2024-03-21T12:54:26.547 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 123 M 2024-03-21T12:54:26.547 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T12:54:26.837 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 337 kB/s | 44 kB 00:00 2024-03-21T12:54:27.847 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-21T12:54:29.245 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 5.9 MB/s | 15 MB 00:02 2024-03-21T12:54:30.820 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 8.1 MB/s | 33 MB 00:04 2024-03-21T12:54:32.362 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.0 MB/s | 27 MB 00:05 2024-03-21T12:54:32.362 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:54:32.362 INFO:teuthology.orchestra.run.smithi089.stdout:Total 13 MB/s | 76 MB 00:05 2024-03-21T12:54:32.711 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-21T12:54:33.031 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T12:54:33.411 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T12:54:33.411 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T12:54:34.420 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T12:54:34.420 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T12:54:34.785 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T12:54:34.785 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T12:54:34.884 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T12:54:34.895 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T12:54:34.895 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-21T12:54:34.895 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T12:54:34.895 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-03-21T12:54:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T12:54:34.897 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 76 M 2024-03-21T12:54:34.897 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 123 M 2024-03-21T12:54:34.897 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T12:54:35.637 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 71 kB/s | 44 kB 00:00 2024-03-21T12:54:36.898 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T12:54:37.492 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:54:37.637 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:54:40.375 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:54:40.516 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:54:45.215 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:54:45.249 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:54:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:54:55.610 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 741 kB/s | 15 MB 00:20 2024-03-21T12:55:12.800 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 757 kB/s | 27 MB 00:37 2024-03-21T12:55:18.525 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 787 kB/s | 33 MB 00:43 2024-03-21T12:55:18.525 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:55:18.525 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.7 MB/s | 76 MB 00:43 2024-03-21T12:55:19.164 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:55:19.164 INFO:teuthology.orchestra.run.smithi089.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T12:55:19.165 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T12:55:19.185 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T12:55:19.208 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:55:19.535 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:55:19.536 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:55:19.536 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:55:19.536 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:55:19.554 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T12:55:19.554 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T12:55:19.943 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T12:55:20.276 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:55:20.378 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:55:20.379 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:55:20.379 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T12:55:20.379 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T12:55:20.379 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T12:55:20.379 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-21T12:55:20.379 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T12:55:20.379 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:55:20.458 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:55:20.459 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:55:20.459 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.06 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 68.el9 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Group : Unspecified 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 8264254 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T12:55:20.501 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : builder@centos.org 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-03-21T12:55:20.502 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T12:55:20.503 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T12:55:20.503 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T12:55:20.503 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 5.14.0-430.el9.x86_64 2024-03-21T12:55:20.504 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T12:55:20.557 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T12:55:20.558 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T12:55:21.142 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-03-21T12:55:22.547 INFO:teuthology.orchestra.run.smithi089.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T12:55:22.588 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-03-21T12:55:22.590 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-03-21T12:55:22.606 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.MWjCsqBMhg 2024-03-21T12:55:22.606 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MWjCsqBMhg 2024-03-21T12:55:22.675 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.MWjCsqBMhg 2024-03-21T12:55:22.815 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.MWjCsqBMhg is 6KB 2024-03-21T12:55:22.863 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.MWjCsqBMhg 2024-03-21T12:55:22.877 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T12:55:22.942 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T12:55:22.942 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T12:55:22.942 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T12:55:22.942 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T12:55:22.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T12:55:23.060 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T12:55:23.089 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-03-21T12:55:23.673 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:55:23.762 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:55:26.506 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:55:26.642 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:55:31.335 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:55:31.442 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:55:36.172 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:55:53.092 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-21T12:55:53.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-21T12:55:53.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:56:05.507 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:56:05.508 INFO:teuthology.orchestra.run.smithi152.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T12:56:05.508 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T12:56:05.552 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:56:05.992 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:56:05.992 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:56:05.992 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:56:05.992 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T12:56:06.420 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T12:56:06.751 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:56:06.861 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:56:06.861 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:56:06.861 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T12:56:06.861 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T12:56:06.861 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T12:56:06.861 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-21T12:56:06.861 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T12:56:06.862 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:56:06.941 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:56:06.941 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:56:06.941 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-03-21T12:56:06.982 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-03-21T12:56:06.982 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-03-21T12:56:06.982 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-03-21T12:56:06.982 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 68.el9 2024-03-21T12:56:06.982 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8264254 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T12:56:06.983 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T12:56:06.984 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T12:56:06.984 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T12:56:06.984 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-03-21T12:56:06.984 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T12:56:06.984 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T12:56:06.985 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T12:56:06.985 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-430.el9.x86_64 2024-03-21T12:56:06.986 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T12:56:07.619 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-03-21T12:56:09.089 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T12:56:09.119 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-03-21T12:56:09.122 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-03-21T12:56:09.137 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.tnYGlMgKpr 2024-03-21T12:56:09.137 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tnYGlMgKpr 2024-03-21T12:56:09.206 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.tnYGlMgKpr 2024-03-21T12:56:09.347 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.tnYGlMgKpr is 6KB 2024-03-21T12:56:09.396 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.tnYGlMgKpr 2024-03-21T12:56:09.410 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T12:56:09.475 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T12:56:09.475 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T12:56:09.475 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T12:56:09.476 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T12:56:09.477 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T12:56:09.618 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-03-21T12:56:27.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-03-21T12:56:36.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-21T12:56:36.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:56:39.624 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-21T12:56:39.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-21T12:56:39.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:56:55.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-03-21T12:57:07.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-21T12:57:07.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:57:07.480 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-03-21T12:57:08.020 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-21T12:57:08.020 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-21T12:57:08.021 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-03-21T12:57:08.035 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-430.el9.x86_64 2024-03-21T12:57:08.035 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T12:57:08.035 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T12:57:08.035 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T12:57:09.036 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-21T12:57:09.036 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T12:57:09.103 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-03-21T12:57:09.154 DEBUG:teuthology.parallel:result is None 2024-03-21T12:57:39.628 DEBUG:teuthology.orchestra.remote:timed out 2024-03-21T12:57:48.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-21T12:57:48.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:57:56.191 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-21T12:57:56.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-21T12:57:56.726 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-21T12:57:56.727 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-21T12:57:56.740 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-430.el9.x86_64 2024-03-21T12:57:56.741 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T12:57:56.741 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T12:57:56.741 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T12:57:57.742 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-21T12:57:57.742 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T12:57:57.810 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-03-21T12:57:57.850 DEBUG:teuthology.parallel:result is None 2024-03-21T12:57:57.850 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-21T12:57:57.858 INFO:teuthology.task.internal:Creating test directory... 2024-03-21T12:57:57.859 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T12:57:57.862 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T12:57:57.878 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-21T12:57:57.885 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-21T12:57:57.895 INFO:teuthology.task.internal:Creating archive directory... 2024-03-21T12:57:57.896 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T12:57:57.920 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T12:57:57.985 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-21T12:57:57.993 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-21T12:57:57.993 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T12:57:57.995 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T12:57:58.036 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:57:58.039 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:57:58.049 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:57:58.052 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:57:58.053 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-21T12:57:58.081 INFO:teuthology.task.internal:Configuring sudo... 2024-03-21T12:57:58.081 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T12:57:58.093 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T12:57:58.121 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-21T12:57:58.129 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-21T12:57:58.129 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T12:57:58.164 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T12:57:58.179 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:57:58.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:57:58.309 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:57:58.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:57:58.434 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T12:57:58.434 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T12:57:58.501 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:57:58.543 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:57:58.593 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:57:58.672 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:57:58.717 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T12:57:58.717 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T12:57:58.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-03-21T12:57:58.787 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-03-21T12:57:58.859 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T12:57:58.876 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T12:57:59.111 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-21T12:57:59.120 INFO:teuthology.task.internal:Starting timer... 2024-03-21T12:57:59.120 INFO:teuthology.run_tasks:Running task pcp... 2024-03-21T12:57:59.133 INFO:teuthology.run_tasks:Running task selinux... 2024-03-21T12:57:59.141 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-21T12:57:59.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-03-21T12:57:59.200 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-03-21T12:57:59.202 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-03-21T12:57:59.250 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-03-21T12:57:59.251 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-21T12:57:59.252 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-03-21T12:57:59.289 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-03-21T12:57:59.289 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-03-21T12:57:59.313 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-03-21T12:57:59.313 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-03-21T12:57:59.314 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T12:57:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:57:59.356 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T12:57:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:57:59.384 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-21T12:57:59.384 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-03-21T12:57:59.432 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-03-21T12:57:59.466 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-21T12:57:59.476 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-21T12:57:59.497 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-21T12:57:59.499 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-21T13:02:15.706 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-03-21T13:02:15.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-21T13:02:15.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:02:15.802 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-03-21T13:02:15.884 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-21T13:02:15.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-21T13:02:15.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:02:15.975 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-21T13:02:16.048 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-21T13:02:16.048 INFO:teuthology.run_tasks:Running task clock... 2024-03-21T13:02:16.061 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-21T13:02:16.061 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T13:02:16.062 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:02:16.067 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T13:02:16.067 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:02:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T13:02:16.111 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T13:02:16.118 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T13:02:16.134 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T13:02:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-03-21T13:02:16.154 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-03-21T13:02:16.163 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-03-21T13:02:16.170 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T13:02:16.177 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-03-21T13:02:16.186 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T13:02:16.194 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T13:02:16.210 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T13:02:16.237 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:02:16.245 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:02:16.245 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-03-21T13:02:16.246 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.246 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.246 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.246 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.269 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:02:16.276 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:02:16.277 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-21T13:02:16.277 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.277 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.277 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.277 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:02:16.278 INFO:teuthology.run_tasks:Running task pexec... 2024-03-21T13:02:16.287 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-21T13:02:16.288 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi089.front.sepia.ceph.com 2024-03-21T13:02:16.288 DEBUG:teuthology.orchestra.run.smithi089:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-21T13:02:16.288 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi152.front.sepia.ceph.com 2024-03-21T13:02:16.288 DEBUG:teuthology.orchestra.run.smithi152:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-21T13:02:16.872 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:46 ago on Thu 21 Mar 2024 01:01:30 PM UTC. 2024-03-21T13:02:16.902 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:53 ago on Thu 21 Mar 2024 01:01:23 PM UTC. 2024-03-21T13:02:17.018 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:02:17.040 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:02:17.041 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:02:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:02:17.050 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:02:17.051 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.1 M 2024-03-21T13:02:17.051 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 9.8 M 2024-03-21T13:02:17.051 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:02:17.051 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:02:17.052 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:02:17.074 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.1 M 2024-03-21T13:02:17.074 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 9.8 M 2024-03-21T13:02:17.074 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:02:17.648 INFO:teuthology.orchestra.run.smithi152.stdout:runc-1.1.12-2.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-03-21T13:02:17.649 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:02:17.649 INFO:teuthology.orchestra.run.smithi152.stdout:Total 5.4 MB/s | 3.1 MB 00:00 2024-03-21T13:02:17.685 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:02:17.691 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:02:17.691 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:02:17.769 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:02:17.770 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:02:17.980 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:02:18.076 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-21T13:02:18.672 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-21T13:02:18.967 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-21T13:02:18.967 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:02:18.967 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:02:18.967 INFO:teuthology.orchestra.run.smithi152.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-21T13:02:18.967 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:02:18.967 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:02:19.201 DEBUG:teuthology.parallel:result is None 2024-03-21T13:02:25.148 INFO:teuthology.orchestra.run.smithi089.stdout:runc-1.1.12-2.el9.x86_64.rpm 402 kB/s | 3.1 MB 00:07 2024-03-21T13:02:25.148 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:02:25.148 INFO:teuthology.orchestra.run.smithi089.stdout:Total 396 kB/s | 3.1 MB 00:08 2024-03-21T13:02:25.187 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:02:25.193 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:02:25.193 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:02:25.275 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:02:25.275 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:02:25.482 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:02:25.586 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-21T13:02:26.208 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-21T13:02:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-21T13:02:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:02:26.515 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:02:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-21T13:02:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:02:26.516 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:02:26.748 DEBUG:teuthology.parallel:result is None 2024-03-21T13:02:26.749 INFO:teuthology.run_tasks:Running task install... 2024-03-21T13:02:26.758 DEBUG:teuthology.task.install:project ceph 2024-03-21T13:02:26.758 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'}} 2024-03-21T13:02:26.758 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'} 2024-03-21T13:02:26.758 INFO:teuthology.task.install:Using flavor: default 2024-03-21T13:02:26.764 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-21T13:02:26.765 INFO:teuthology.task.install:extra packages: [] 2024-03-21T13:02:26.765 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'tag': None, 'wait_for_package': False} 2024-03-21T13:02:26.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:02:26.768 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'tag': None, 'wait_for_package': False} 2024-03-21T13:02:26.768 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:02:26.915 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/ 2024-03-21T13:02:26.915 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2267.g960fbbe4 2024-03-21T13:02:26.987 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/ 2024-03-21T13:02:26.987 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2267.g960fbbe4 2024-03-21T13:02:27.086 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T13:02:27.086 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:02:27.086 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T13:02:27.114 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-21T13:02:27.114 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/960fbbe4645c2becc8084669be194c95183c7753/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T13:02:27.159 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T13:02:27.159 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:02:27.159 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T13:02:27.181 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T13:02:27.191 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-21T13:02:27.192 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/960fbbe4645c2becc8084669be194c95183c7753/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T13:02:27.261 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T13:02:27.263 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T13:02:27.328 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-03-21T13:02:27.329 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-03-21T13:02:27.347 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T13:02:27.415 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-03-21T13:02:27.417 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-03-21T13:02:27.583 INFO:teuthology.orchestra.run.smithi089.stdout:74 files removed 2024-03-21T13:02:27.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2024-03-21T13:02:27.656 INFO:teuthology.orchestra.run.smithi152.stdout:74 files removed 2024-03-21T13:02:27.683 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-03-21T13:02:28.270 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 231 kB/s | 83 kB 00:00 2024-03-21T13:02:28.337 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 238 kB/s | 83 kB 00:00 2024-03-21T13:02:28.671 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 41 kB/s | 12 kB 00:00 2024-03-21T13:02:28.704 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-03-21T13:02:28.972 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 7.3 kB/s | 1.9 kB 00:00 2024-03-21T13:02:29.113 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-21T13:02:29.366 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-21T13:02:29.549 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-21T13:02:30.791 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-21T13:02:30.941 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-21T13:02:32.007 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-03-21T13:02:32.132 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.0 MB 00:00 2024-03-21T13:02:33.775 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-21T13:02:34.366 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-21T13:02:38.867 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.1 MB 00:00 2024-03-21T13:02:40.617 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-21T13:02:41.811 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-21T13:02:46.777 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-21T13:02:49.194 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:02:49.196 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:02:49.196 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:02:49.196 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:02:49.196 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 9.9 M 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.4 M 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.4 M 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 20 M 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 25 k 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T13:02:49.197 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 746 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 501 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.1 M 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 45 k 2024-03-21T13:02:49.198 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 131 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 162 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 320 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 298 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 99 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:02:49.199 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:02:49.200 INFO:teuthology.orchestra.run.smithi089.stdout:Install 30 Packages 2024-03-21T13:02:49.200 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2024-03-21T13:02:49.200 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:02:49.200 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 59 M 2024-03-21T13:02:49.200 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:02:50.120 INFO:teuthology.orchestra.run.smithi089.stdout:(1/32): ceph-base-19.0.0-2267.g960fbbe4.el9.x86 7.7 MB/s | 5.4 MB 00:00 2024-03-21T13:02:50.187 INFO:teuthology.orchestra.run.smithi089.stdout:(2/32): ceph-selinux-19.0.0-2267.g960fbbe4.el9. 376 kB/s | 25 kB 00:00 2024-03-21T13:02:50.287 INFO:teuthology.orchestra.run.smithi089.stdout:(3/32): libcephfs2-19.0.0-2267.g960fbbe4.el9.x8 7.3 MB/s | 746 kB 00:00 2024-03-21T13:02:50.358 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 480 kB/s | 6.1 MB 00:13 2024-03-21T13:02:50.396 INFO:teuthology.orchestra.run.smithi089.stdout:(4/32): libradosstriper1-19.0.0-2267.g960fbbe4. 4.5 MB/s | 501 kB 00:00 2024-03-21T13:02:50.863 INFO:teuthology.orchestra.run.smithi089.stdout:(5/32): ceph-radosgw-19.0.0-2267.g960fbbe4.el9. 6.8 MB/s | 9.9 MB 00:01 2024-03-21T13:02:50.938 INFO:teuthology.orchestra.run.smithi089.stdout:(6/32): python3-ceph-argparse-19.0.0-2267.g960f 604 kB/s | 45 kB 00:00 2024-03-21T13:02:51.005 INFO:teuthology.orchestra.run.smithi089.stdout:(7/32): librgw2-19.0.0-2267.g960fbbe4.el9.x86_6 8.5 MB/s | 5.1 MB 00:00 2024-03-21T13:02:51.030 INFO:teuthology.orchestra.run.smithi089.stdout:(8/32): python3-ceph-common-19.0.0-2267.g960fbb 1.4 MB/s | 131 kB 00:00 2024-03-21T13:02:51.064 INFO:teuthology.orchestra.run.smithi089.stdout:(9/32): python3-cephfs-19.0.0-2267.g960fbbe4.el 2.7 MB/s | 162 kB 00:00 2024-03-21T13:02:51.097 INFO:teuthology.orchestra.run.smithi089.stdout:(10/32): python3-rados-19.0.0-2267.g960fbbe4.el 4.7 MB/s | 320 kB 00:00 2024-03-21T13:02:51.131 INFO:teuthology.orchestra.run.smithi089.stdout:(11/32): python3-rbd-19.0.0-2267.g960fbbe4.el9. 4.4 MB/s | 298 kB 00:00 2024-03-21T13:02:51.332 INFO:teuthology.orchestra.run.smithi089.stdout:(12/32): ceph-common-19.0.0-2267.g960fbbe4.el9. 11 MB/s | 20 MB 00:01 2024-03-21T13:02:51.382 INFO:teuthology.orchestra.run.smithi089.stdout:(13/32): python3-rgw-19.0.0-2267.g960fbbe4.el9. 350 kB/s | 99 kB 00:00 2024-03-21T13:02:51.449 INFO:teuthology.orchestra.run.smithi089.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 333 kB/s | 106 kB 00:00 2024-03-21T13:02:51.466 INFO:teuthology.orchestra.run.smithi089.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 746 kB/s | 44 kB 00:00 2024-03-21T13:02:51.499 INFO:teuthology.orchestra.run.smithi089.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-21T13:02:51.575 INFO:teuthology.orchestra.run.smithi089.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 661 kB 00:00 2024-03-21T13:02:51.600 INFO:teuthology.orchestra.run.smithi089.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-21T13:02:51.616 INFO:teuthology.orchestra.run.smithi089.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 351 kB/s | 41 kB 00:00 2024-03-21T13:02:51.701 INFO:teuthology.orchestra.run.smithi089.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 419 kB/s | 35 kB 00:00 2024-03-21T13:02:51.717 INFO:teuthology.orchestra.run.smithi089.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 444 kB/s | 44 kB 00:00 2024-03-21T13:02:51.776 INFO:teuthology.orchestra.run.smithi089.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-21T13:02:51.851 INFO:teuthology.orchestra.run.smithi089.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-21T13:02:51.876 INFO:teuthology.orchestra.run.smithi089.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-03-21T13:02:51.901 INFO:teuthology.orchestra.run.smithi089.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-03-21T13:02:51.927 INFO:teuthology.orchestra.run.smithi089.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-21T13:02:51.933 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-21T13:02:51.977 INFO:teuthology.orchestra.run.smithi089.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-21T13:02:52.052 INFO:teuthology.orchestra.run.smithi089.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-21T13:02:52.086 INFO:teuthology.orchestra.run.smithi089.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 838 kB 00:00 2024-03-21T13:02:52.153 INFO:teuthology.orchestra.run.smithi089.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-21T13:02:52.420 INFO:teuthology.orchestra.run.smithi089.stdout:(31/32): librbd1-19.0.0-2267.g960fbbe4.el9.x86_ 9.4 MB/s | 3.1 MB 00:00 2024-03-21T13:02:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:(32/32): librados2-19.0.0-2267.g960fbbe4.el9.x8 8.0 MB/s | 3.4 MB 00:00 2024-03-21T13:02:52.481 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:02:52.481 INFO:teuthology.orchestra.run.smithi089.stdout:Total 18 MB/s | 59 MB 00:03 2024-03-21T13:02:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:02:52.548 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:02:52.548 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:02:53.042 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:02:53.042 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:02:53.585 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-21T13:02:54.473 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:02:54.596 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-21T13:02:54.625 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-21T13:02:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-21T13:02:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:02:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:02:54.849 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:02:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:02:55.230 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-21T13:02:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T13:02:55.377 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T13:02:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 8/34 2024-03-21T13:02:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T13:02:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T13:02:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-21T13:02:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 11/34 2024-03-21T13:02:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T13:02:55.722 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T13:02:55.771 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T13:02:55.804 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-21T13:02:55.915 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-21T13:02:55.955 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-21T13:02:55.996 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-21T13:02:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-21T13:02:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-21T13:02:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-21T13:02:56.414 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-21T13:02:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T13:02:56.536 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T13:02:56.566 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 23/34 2024-03-21T13:02:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-21T13:02:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-21T13:02:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T13:02:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T13:02:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 27/34 2024-03-21T13:02:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 28/34 2024-03-21T13:02:58.635 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-21T13:02:59.419 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T13:02:59.455 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T13:03:00.009 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T13:03:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T13:03:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T13:03:00.115 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T13:03:00.115 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T13:03:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T13:03:01.061 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 9.9 M 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.4 M 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T13:03:01.064 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.4 M 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 20 M 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 25 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 746 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T13:03:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 501 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 5.1 M 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 45 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 131 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 162 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T13:03:01.066 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 320 k 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 298 k 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 99 k 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout:Install 30 Packages 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-03-21T13:03:01.067 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:01.068 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 59 M 2024-03-21T13:03:01.068 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:03:02.394 INFO:teuthology.orchestra.run.smithi152.stdout:(1/32): ceph-base-19.0.0-2267.g960fbbe4.el9.x86 5.3 MB/s | 5.4 MB 00:01 2024-03-21T13:03:02.453 INFO:teuthology.orchestra.run.smithi152.stdout:(2/32): ceph-selinux-19.0.0-2267.g960fbbe4.el9. 433 kB/s | 25 kB 00:00 2024-03-21T13:03:02.603 INFO:teuthology.orchestra.run.smithi152.stdout:(3/32): libcephfs2-19.0.0-2267.g960fbbe4.el9.x8 4.9 MB/s | 746 kB 00:00 2024-03-21T13:03:02.745 INFO:teuthology.orchestra.run.smithi152.stdout:(4/32): libradosstriper1-19.0.0-2267.g960fbbe4. 3.5 MB/s | 501 kB 00:00 2024-03-21T13:03:03.662 INFO:teuthology.orchestra.run.smithi152.stdout:(5/32): librgw2-19.0.0-2267.g960fbbe4.el9.x86_6 5.6 MB/s | 5.1 MB 00:00 2024-03-21T13:03:03.737 INFO:teuthology.orchestra.run.smithi152.stdout:(6/32): python3-ceph-argparse-19.0.0-2267.g960f 603 kB/s | 45 kB 00:00 2024-03-21T13:03:03.829 INFO:teuthology.orchestra.run.smithi152.stdout:(7/32): python3-ceph-common-19.0.0-2267.g960fbb 1.4 MB/s | 131 kB 00:00 2024-03-21T13:03:03.930 INFO:teuthology.orchestra.run.smithi152.stdout:(8/32): python3-cephfs-19.0.0-2267.g960fbbe4.el 1.6 MB/s | 162 kB 00:00 2024-03-21T13:03:04.038 INFO:teuthology.orchestra.run.smithi152.stdout:(9/32): python3-rados-19.0.0-2267.g960fbbe4.el9 2.9 MB/s | 320 kB 00:00 2024-03-21T13:03:04.122 INFO:teuthology.orchestra.run.smithi152.stdout:(10/32): python3-rbd-19.0.0-2267.g960fbbe4.el9. 3.5 MB/s | 298 kB 00:00 2024-03-21T13:03:04.189 INFO:teuthology.orchestra.run.smithi152.stdout:(11/32): python3-rgw-19.0.0-2267.g960fbbe4.el9. 1.5 MB/s | 99 kB 00:00 2024-03-21T13:03:04.289 INFO:teuthology.orchestra.run.smithi152.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-21T13:03:04.339 INFO:teuthology.orchestra.run.smithi152.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-21T13:03:04.373 INFO:teuthology.orchestra.run.smithi152.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-21T13:03:04.607 INFO:teuthology.orchestra.run.smithi152.stdout:(15/32): ceph-common-19.0.0-2267.g960fbbe4.el9. 6.4 MB/s | 20 MB 00:03 2024-03-21T13:03:04.632 INFO:teuthology.orchestra.run.smithi152.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.5 MB/s | 661 kB 00:00 2024-03-21T13:03:04.829 INFO:teuthology.orchestra.run.smithi152.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.2 MB/s | 282 kB 00:00 2024-03-21T13:03:04.875 INFO:teuthology.orchestra.run.smithi152.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 170 kB/s | 41 kB 00:00 2024-03-21T13:03:04.992 INFO:teuthology.orchestra.run.smithi152.stdout:(19/32): ceph-radosgw-19.0.0-2267.g960fbbe4.el9 2.7 MB/s | 9.9 MB 00:03 2024-03-21T13:03:05.059 INFO:teuthology.orchestra.run.smithi152.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 190 kB/s | 35 kB 00:00 2024-03-21T13:03:05.109 INFO:teuthology.orchestra.run.smithi152.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 381 kB/s | 44 kB 00:00 2024-03-21T13:03:05.134 INFO:teuthology.orchestra.run.smithi152.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 712 kB/s | 216 kB 00:00 2024-03-21T13:03:05.193 INFO:teuthology.orchestra.run.smithi152.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.3 MB/s | 309 kB 00:00 2024-03-21T13:03:05.210 INFO:teuthology.orchestra.run.smithi152.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-21T13:03:05.226 INFO:teuthology.orchestra.run.smithi152.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-21T13:03:05.269 INFO:teuthology.orchestra.run.smithi152.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-21T13:03:05.302 INFO:teuthology.orchestra.run.smithi152.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-21T13:03:05.328 INFO:teuthology.orchestra.run.smithi152.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-21T13:03:05.411 INFO:teuthology.orchestra.run.smithi152.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-21T13:03:05.453 INFO:teuthology.orchestra.run.smithi152.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-21T13:03:06.187 INFO:teuthology.orchestra.run.smithi152.stdout:(31/32): librados2-19.0.0-2267.g960fbbe4.el9.x8 4.0 MB/s | 3.4 MB 00:00 2024-03-21T13:03:06.245 INFO:teuthology.orchestra.run.smithi152.stdout:(32/32): librbd1-19.0.0-2267.g960fbbe4.el9.x86_ 3.8 MB/s | 3.1 MB 00:00 2024-03-21T13:03:06.247 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:06.248 INFO:teuthology.orchestra.run.smithi152.stdout:Total 11 MB/s | 59 MB 00:05 2024-03-21T13:03:06.282 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:03:06.320 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:03:06.320 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:03:06.825 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:03:06.826 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:03:08.285 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:03:08.373 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-21T13:03:08.400 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-21T13:03:08.426 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-21T13:03:08.448 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:03:08.464 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:03:08.627 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:03:08.687 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:03:09.025 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-21T13:03:09.086 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T13:03:09.178 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T13:03:09.268 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 8/34 2024-03-21T13:03:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T13:03:09.359 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 9/34 2024-03-21T13:03:09.388 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-21T13:03:09.433 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 11/34 2024-03-21T13:03:09.502 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T13:03:09.523 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T13:03:09.582 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 13/34 2024-03-21T13:03:09.614 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-21T13:03:09.724 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-21T13:03:09.765 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-21T13:03:09.805 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-21T13:03:09.838 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-21T13:03:09.867 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-21T13:03:09.900 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-21T13:03:10.065 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T13:03:10.065 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-03-21T13:03:10.065 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-03-21T13:03:10.065 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-03-21T13:03:10.065 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-03-21T13:03:10.066 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-03-21T13:03:10.066 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-03-21T13:03:10.066 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-03-21T13:03:10.066 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:10.226 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-21T13:03:10.255 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T13:03:10.346 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 22/34 2024-03-21T13:03:10.376 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 23/34 2024-03-21T13:03:10.416 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-21T13:03:10.878 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-21T13:03:10.908 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T13:03:10.962 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 26/34 2024-03-21T13:03:11.008 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T13:03:11.031 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 27/34 2024-03-21T13:03:11.033 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T13:03:11.033 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:11.033 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T13:03:11.033 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:03:11.034 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:03:11.034 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:11.035 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:03:11.060 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 28/34 2024-03-21T13:03:11.119 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:03:11.119 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 6/34 2024-03-21T13:03:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 8/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 9/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 10/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 11/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 13/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-21T13:03:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-21T13:03:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-21T13:03:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T13:03:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-21T13:03:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/34 2024-03-21T13:03:13.354 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T13:03:13.382 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T13:03:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T13:03:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:13.402 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:13.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2024-03-21T13:03:13.959 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 29/34 2024-03-21T13:03:14.002 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T13:03:14.102 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 30/34 2024-03-21T13:03:14.102 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T13:03:14.102 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T13:03:14.102 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:14.137 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T13:03:14.294 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:28 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:14.442 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:14.442 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 48 M 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 163 k 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:14.443 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-03-21T13:03:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:14.444 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 49 M 2024-03-21T13:03:14.444 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 206 M 2024-03-21T13:03:14.444 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:03:14.882 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-21T13:03:14.931 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-21T13:03:14.965 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-21T13:03:15.007 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): libcephsqlite-19.0.0-2267.g960fbbe4.el9. 579 kB/s | 163 kB 00:00 2024-03-21T13:03:15.101 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-21T13:03:23.235 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): ceph-test-19.0.0-2267.g960fbbe4.el9.x86_ 5.6 MB/s | 48 MB 00:08 2024-03-21T13:03:23.237 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:23.238 INFO:teuthology.orchestra.run.smithi089.stdout:Total 5.5 MB/s | 49 MB 00:08 2024-03-21T13:03:23.252 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:03:23.273 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:03:23.273 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:03:23.660 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:03:23.660 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:03:23.984 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T13:03:23.984 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-03-21T13:03:23.984 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-03-21T13:03:23.984 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-03-21T13:03:23.985 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-03-21T13:03:23.985 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-03-21T13:03:23.985 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-03-21T13:03:23.985 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-03-21T13:03:23.985 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:24.244 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:03:24.318 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-21T13:03:24.359 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-21T13:03:24.399 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:03:24.443 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-21T13:03:24.471 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T13:03:24.913 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T13:03:24.939 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/34 2024-03-21T13:03:24.939 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:24.939 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T13:03:24.939 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:03:24.939 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:03:24.939 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:24.941 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:03:25.025 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:03:25.025 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:03:26.182 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_6 6/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 7/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9. 8/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x8 9/34 2024-03-21T13:03:26.183 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 10/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 11/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 12/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 13/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-21T13:03:26.184 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-21T13:03:26.185 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 31/34 2024-03-21T13:03:26.186 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-21T13:03:26.186 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/34 2024-03-21T13:03:27.179 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:03:27.179 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:27.179 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-03-21T13:03:27.179 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.180 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T13:03:27.181 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:27.182 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:03:27.556 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-03-21T13:03:28.117 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:30 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:03:28.270 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:03:28.271 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:28.271 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:03:28.271 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:28.271 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:03:28.271 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 48 M 2024-03-21T13:03:28.271 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 163 k 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:28.272 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-03-21T13:03:28.273 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:28.273 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 49 M 2024-03-21T13:03:28.273 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 206 M 2024-03-21T13:03:28.273 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:03:28.553 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-21T13:03:28.602 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-21T13:03:28.636 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-21T13:03:28.703 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): libcephsqlite-19.0.0-2267.g960fbbe4.el9. 538 kB/s | 163 kB 00:00 2024-03-21T13:03:28.762 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-21T13:03:29.005 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T13:03:29.104 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T13:03:30.820 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): ceph-test-19.0.0-2267.g960fbbe4.el9.x86_ 20 MB/s | 48 MB 00:02 2024-03-21T13:03:30.823 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:30.824 INFO:teuthology.orchestra.run.smithi152.stdout:Total 19 MB/s | 49 MB 00:02 2024-03-21T13:03:30.839 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:03:30.860 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:03:30.860 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:03:31.238 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:03:31.238 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:03:31.493 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T13:03:31.493 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/6 2024-03-21T13:03:31.493 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/6 2024-03-21T13:03:31.493 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:03:31.493 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-21T13:03:31.493 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-21T13:03:31.815 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:03:31.898 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-21T13:03:31.937 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-21T13:03:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-16.el9.x86_64 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:31.940 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:31.976 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:03:32.020 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-21T13:03:32.048 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T13:03:32.224 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2024-03-21T13:03:32.790 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:46 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:32.942 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:32.945 INFO:teuthology.orchestra.run.smithi089.stdout:========================================================================================= 2024-03-21T13:03:32.945 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:03:32.945 INFO:teuthology.orchestra.run.smithi089.stdout:========================================================================================= 2024-03-21T13:03:32.945 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:03:32.945 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 6.6 k 2024-03-21T13:03:32.945 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:03:32.945 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 2.3 M 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 1.5 M 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 248 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 4.7 M 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 17 M 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T13:03:32.946 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T13:03:32.947 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout:========================================================================================= 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout:Install 36 Packages 2024-03-21T13:03:32.948 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:32.949 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 30 M 2024-03-21T13:03:32.949 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 107 M 2024-03-21T13:03:32.949 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:03:33.540 INFO:teuthology.orchestra.run.smithi089.stdout:(1/36): ceph-19.0.0-2267.g960fbbe4.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-21T13:03:34.098 INFO:teuthology.orchestra.run.smithi089.stdout:(2/36): ceph-mgr-19.0.0-2267.g960fbbe4.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-03-21T13:03:34.215 INFO:teuthology.orchestra.run.smithi089.stdout:(3/36): ceph-mon-19.0.0-2267.g960fbbe4.el9.x86_ 6.9 MB/s | 4.7 MB 00:00 2024-03-21T13:03:34.266 INFO:teuthology.orchestra.run.smithi089.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2267.g960f 4.8 MB/s | 248 kB 00:00 2024-03-21T13:03:34.349 INFO:teuthology.orchestra.run.smithi089.stdout:(5/36): ceph-mds-19.0.0-2267.g960fbbe4.el9.x86_ 2.3 MB/s | 2.3 MB 00:00 2024-03-21T13:03:34.416 INFO:teuthology.orchestra.run.smithi089.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-03-21T13:03:34.441 INFO:teuthology.orchestra.run.smithi089.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-21T13:03:34.467 INFO:teuthology.orchestra.run.smithi089.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-03-21T13:03:34.492 INFO:teuthology.orchestra.run.smithi089.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-21T13:03:34.517 INFO:teuthology.orchestra.run.smithi089.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 768 kB/s | 39 kB 00:00 2024-03-21T13:03:34.543 INFO:teuthology.orchestra.run.smithi089.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-21T13:03:34.593 INFO:teuthology.orchestra.run.smithi089.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-21T13:03:34.618 INFO:teuthology.orchestra.run.smithi089.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-21T13:03:34.643 INFO:teuthology.orchestra.run.smithi089.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-21T13:03:34.669 INFO:teuthology.orchestra.run.smithi089.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-21T13:03:34.786 INFO:teuthology.orchestra.run.smithi089.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 881 kB/s | 125 kB 00:00 2024-03-21T13:03:35.253 INFO:teuthology.orchestra.run.smithi089.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-03-21T13:03:35.340 INFO:teuthology.orchestra.run.smithi089.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 273 kB/s | 151 kB 00:00 2024-03-21T13:03:35.529 INFO:teuthology.orchestra.run.smithi089.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-21T13:03:35.654 INFO:teuthology.orchestra.run.smithi089.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 108 kB/s | 43 kB 00:00 2024-03-21T13:03:35.712 INFO:teuthology.orchestra.run.smithi089.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-03-21T13:03:35.879 INFO:teuthology.orchestra.run.smithi089.stdout:(22/36): ceph-osd-19.0.0-2267.g960fbbe4.el9.x86 9.5 MB/s | 17 MB 00:01 2024-03-21T13:03:35.896 INFO:teuthology.orchestra.run.smithi089.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 44 kB/s | 11 kB 00:00 2024-03-21T13:03:35.913 INFO:teuthology.orchestra.run.smithi089.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 88 kB/s | 18 kB 00:00 2024-03-21T13:03:35.997 INFO:teuthology.orchestra.run.smithi089.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-21T13:03:36.039 INFO:teuthology.orchestra.run.smithi089.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 145 kB/s | 23 kB 00:00 2024-03-21T13:03:36.131 INFO:teuthology.orchestra.run.smithi089.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 428 kB/s | 46 kB 00:00 2024-03-21T13:03:36.239 INFO:teuthology.orchestra.run.smithi089.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 394 kB/s | 79 kB 00:00 2024-03-21T13:03:36.265 INFO:teuthology.orchestra.run.smithi089.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-21T13:03:36.282 INFO:teuthology.orchestra.run.smithi089.stdout:(30/36): python3-jaraco-text-3.2.0-6.el9.noarch 53 kB/s | 20 kB 00:00 2024-03-21T13:03:36.341 INFO:teuthology.orchestra.run.smithi089.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 162 kB/s | 16 kB 00:00 2024-03-21T13:03:36.366 INFO:teuthology.orchestra.run.smithi089.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 892 kB/s | 90 kB 00:00 2024-03-21T13:03:36.458 INFO:teuthology.orchestra.run.smithi089.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 203 kB/s | 36 kB 00:00 2024-03-21T13:03:36.500 INFO:teuthology.orchestra.run.smithi089.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-21T13:03:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-03-21T13:03:36.575 INFO:teuthology.orchestra.run.smithi089.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-03-21T13:03:36.581 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:36.581 INFO:teuthology.orchestra.run.smithi089.stdout:Total 8.2 MB/s | 30 MB 00:03 2024-03-21T13:03:36.605 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:03:36.643 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/6 2024-03-21T13:03:36.647 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:03:36.648 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:03:36.713 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T13:03:37.055 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:03:37.055 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:03:38.390 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:03:38.470 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-21T13:03:38.511 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-21T13:03:38.530 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-21T13:03:38.605 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-21T13:03:38.638 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-21T13:03:38.682 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-21T13:03:38.763 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-21T13:03:38.841 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-21T13:03:38.883 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-21T13:03:38.954 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-21T13:03:39.007 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-21T13:03:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-21T13:03:39.094 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-21T13:03:39.151 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-21T13:03:39.204 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 6/6 2024-03-21T13:03:39.204 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/6 2024-03-21T13:03:39.204 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/6 2024-03-21T13:03:39.204 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:03:39.204 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-21T13:03:39.204 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-21T13:03:39.334 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-21T13:03:39.375 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-21T13:03:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-21T13:03:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-21T13:03:39.487 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-21T13:03:39.563 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-21T13:03:39.598 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:03:39.607 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-21T13:03:39.607 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:39.607 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout: jq-1.6-16.el9.x86_64 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:39.608 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:03:39.631 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-21T13:03:39.671 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-21T13:03:39.707 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-21T13:03:39.741 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-21T13:03:39.830 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-21T13:03:39.856 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-03-21T13:03:39.864 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-21T13:03:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-21T13:03:40.072 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-21T13:03:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:03:40.351 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 31/36 2024-03-21T13:03:40.378 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T13:03:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T13:03:40.403 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:40.403 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T13:03:40.403 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:03:40.403 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:03:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:40.442 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:42 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:03:40.613 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:03:40.616 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================= 2024-03-21T13:03:40.616 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:03:40.616 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================= 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 6.6 k 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 2.3 M 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 1.5 M 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 248 k 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 4.7 M 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 17 M 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T13:03:40.617 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T13:03:40.618 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================= 2024-03-21T13:03:40.619 INFO:teuthology.orchestra.run.smithi152.stdout:Install 36 Packages 2024-03-21T13:03:40.620 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:40.620 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 30 M 2024-03-21T13:03:40.620 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 107 M 2024-03-21T13:03:40.620 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:03:41.296 INFO:teuthology.orchestra.run.smithi152.stdout:(1/36): ceph-19.0.0-2267.g960fbbe4.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-21T13:03:41.505 INFO:teuthology.orchestra.run.smithi152.stdout:(2/36): ceph-mds-19.0.0-2267.g960fbbe4.el9.x86_ 5.1 MB/s | 2.3 MB 00:00 2024-03-21T13:03:41.622 INFO:teuthology.orchestra.run.smithi152.stdout:(3/36): ceph-mgr-19.0.0-2267.g960fbbe4.el9.x86_ 2.6 MB/s | 1.5 MB 00:00 2024-03-21T13:03:41.705 INFO:teuthology.orchestra.run.smithi152.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2267.g960f 2.9 MB/s | 248 kB 00:00 2024-03-21T13:03:41.854 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T13:03:41.864 INFO:teuthology.orchestra.run.smithi152.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-21T13:03:41.880 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T13:03:41.880 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:41.881 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T13:03:41.881 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:03:41.881 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:03:41.881 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:41.906 INFO:teuthology.orchestra.run.smithi152.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-21T13:03:41.940 INFO:teuthology.orchestra.run.smithi152.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-21T13:03:41.973 INFO:teuthology.orchestra.run.smithi152.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-21T13:03:42.007 INFO:teuthology.orchestra.run.smithi152.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-21T13:03:42.066 INFO:teuthology.orchestra.run.smithi152.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-21T13:03:42.166 INFO:teuthology.orchestra.run.smithi152.stdout:(11/36): ceph-mon-19.0.0-2267.g960fbbe4.el9.x86 5.4 MB/s | 4.7 MB 00:00 2024-03-21T13:03:42.199 INFO:teuthology.orchestra.run.smithi152.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-03-21T13:03:42.225 INFO:teuthology.orchestra.run.smithi152.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-21T13:03:42.250 INFO:teuthology.orchestra.run.smithi152.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-21T13:03:42.275 INFO:teuthology.orchestra.run.smithi152.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-21T13:03:42.330 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T13:03:42.353 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T13:03:42.353 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:42.353 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T13:03:42.353 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:03:42.353 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:03:42.353 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:42.376 INFO:teuthology.orchestra.run.smithi152.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 996 kB/s | 125 kB 00:00 2024-03-21T13:03:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T13:03:42.576 INFO:teuthology.orchestra.run.smithi152.stdout:(17/36): ceph-osd-19.0.0-2267.g960fbbe4.el9.x86 16 MB/s | 17 MB 00:01 2024-03-21T13:03:42.595 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T13:03:42.595 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:42.595 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T13:03:42.595 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:03:42.595 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:03:42.595 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T13:03:42.709 INFO:teuthology.orchestra.run.smithi152.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 51 kB/s | 22 kB 00:00 2024-03-21T13:03:42.734 INFO:teuthology.orchestra.run.smithi152.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 275 kB/s | 43 kB 00:00 2024-03-21T13:03:42.760 INFO:teuthology.orchestra.run.smithi152.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 393 kB/s | 151 kB 00:00 2024-03-21T13:03:42.794 INFO:teuthology.orchestra.run.smithi152.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-21T13:03:42.819 INFO:teuthology.orchestra.run.smithi152.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 182 kB/s | 11 kB 00:00 2024-03-21T13:03:42.844 INFO:teuthology.orchestra.run.smithi152.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-21T13:03:42.869 INFO:teuthology.orchestra.run.smithi152.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-21T13:03:42.894 INFO:teuthology.orchestra.run.smithi152.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-21T13:03:42.948 INFO:teuthology.orchestra.run.smithi152.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 251 kB/s | 20 kB 00:00 2024-03-21T13:03:42.970 INFO:teuthology.orchestra.run.smithi152.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-03-21T13:03:43.003 INFO:teuthology.orchestra.run.smithi152.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-21T13:03:43.029 INFO:teuthology.orchestra.run.smithi152.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 977 kB/s | 79 kB 00:00 2024-03-21T13:03:43.071 INFO:teuthology.orchestra.run.smithi152.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-21T13:03:43.104 INFO:teuthology.orchestra.run.smithi152.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-21T13:03:43.138 INFO:teuthology.orchestra.run.smithi152.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-21T13:03:43.163 INFO:teuthology.orchestra.run.smithi152.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-03-21T13:03:43.230 INFO:teuthology.orchestra.run.smithi152.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-21T13:03:43.255 INFO:teuthology.orchestra.run.smithi152.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-21T13:03:43.356 INFO:teuthology.orchestra.run.smithi152.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-03-21T13:03:43.358 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:43.358 INFO:teuthology.orchestra.run.smithi152.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-21T13:03:43.383 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:03:43.426 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:03:43.426 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:03:43.843 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:03:43.843 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:03:44.500 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 6/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-21T13:03:44.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-21T13:03:44.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-21T13:03:44.503 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-21T13:03:44.503 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-21T13:03:44.504 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:03:44.504 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-21T13:03:44.504 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-21T13:03:44.504 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-21T13:03:44.504 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-21T13:03:44.504 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-21T13:03:45.148 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:03:45.233 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-21T13:03:45.266 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-21T13:03:45.294 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-21T13:03:45.366 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-21T13:03:45.450 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-21T13:03:45.483 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-21T13:03:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T13:03:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T13:03:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:45.537 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:45.583 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-21T13:03:45.670 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-21T13:03:45.711 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-21T13:03:45.784 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-21T13:03:45.845 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-21T13:03:45.883 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-21T13:03:45.896 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2024-03-21T13:03:45.940 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-21T13:03:45.996 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-21T13:03:46.181 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-21T13:03:46.229 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-21T13:03:46.272 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-21T13:03:46.310 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-21T13:03:46.348 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-21T13:03:46.425 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-21T13:03:46.459 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:03:46.460 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:00 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:46.492 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-21T13:03:46.532 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-21T13:03:46.558 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:03:46.567 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-21T13:03:46.601 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-21T13:03:46.609 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:46.610 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:03:46.610 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:46.685 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2024-03-21T13:03:46.692 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-21T13:03:46.733 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-21T13:03:46.805 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-21T13:03:46.967 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-21T13:03:47.084 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:03:47.239 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:01 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:47.250 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 31/36 2024-03-21T13:03:47.287 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T13:03:47.313 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 32/36 2024-03-21T13:03:47.313 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:47.313 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T13:03:47.313 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:03:47.313 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:03:47.313 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:47.386 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:47.387 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:03:47.387 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:47.387 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 750 k 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 750 k 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 757 k 2024-03-21T13:03:47.388 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:03:47.658 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-19.0.0-2267.g960fbbe4.el9.noarch.rpm 2.7 MB/s | 750 kB 00:00 2024-03-21T13:03:47.659 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:47.659 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.7 MB/s | 750 kB 00:00 2024-03-21T13:03:47.659 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:03:47.664 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:03:47.664 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:03:47.669 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:03:47.670 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:03:47.742 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:03:48.099 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:48.179 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:48.553 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:48.762 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T13:03:48.789 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 33/36 2024-03-21T13:03:48.789 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:48.789 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T13:03:48.789 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:03:48.789 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:03:48.789 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:48.819 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:48.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:48.820 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:03:48.820 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:03:48.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:48.820 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:49.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2024-03-21T13:03:49.246 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T13:03:49.270 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 34/36 2024-03-21T13:03:49.270 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:49.270 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T13:03:49.270 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:03:49.270 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:03:49.270 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T13:03:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 35/36 2024-03-21T13:03:49.526 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:49.526 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T13:03:49.526 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:03:49.526 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:03:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:49.593 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T13:03:49.613 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:03 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:49.767 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:49.767 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 145 k 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 145 k 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 436 k 2024-03-21T13:03:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:03:50.115 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-19.0.0-2267.g960fbb 418 kB/s | 145 kB 00:00 2024-03-21T13:03:50.116 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:50.116 INFO:teuthology.orchestra.run.smithi089.stdout:Total 417 kB/s | 145 kB 00:00 2024-03-21T13:03:50.116 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:03:50.124 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:03:50.124 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:03:50.176 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:03:50.176 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:03:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:03:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T13:03:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T13:03:50.416 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:50.416 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T13:03:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:51.458 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 36/36 2024-03-21T13:03:51.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/36 2024-03-21T13:03:51.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/36 2024-03-21T13:03:51.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 3/36 2024-03-21T13:03:51.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 4/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 5/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9. 6/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-21T13:03:51.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-21T13:03:51.460 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-21T13:03:51.460 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-21T13:03:51.460 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:03:51.460 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-21T13:03:51.460 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-21T13:03:51.460 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-21T13:03:51.460 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-21T13:03:51.461 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-21T13:03:51.462 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-21T13:03:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T13:03:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:51.479 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:03:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:51.479 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:51.729 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2024-03-21T13:03:52.304 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:06 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:52.395 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-21T13:03:52.395 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T13:03:52.396 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T13:03:52.397 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:52.398 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:03:52.400 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:03:52.451 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:52.452 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:03:52.452 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:52.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2024-03-21T13:03:52.748 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-03-21T13:03:53.078 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:07 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:53.226 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:53.227 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-03-21T13:03:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:03:53.227 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 3.5 M 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 34 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 16 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T13:03:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T13:03:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:53.229 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:03:53.229 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-03-21T13:03:53.229 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-03-21T13:03:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:53.229 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 8.6 M 2024-03-21T13:03:53.229 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 103 M 2024-03-21T13:03:53.230 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:03:53.330 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:55 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:03:53.427 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:03:53.478 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:03:53.479 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:03:53.479 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:03:53.545 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-03-21T13:03:53.896 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2267.g960 86 kB/s | 16 kB 00:00 2024-03-21T13:03:53.946 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2267.g96 140 kB/s | 34 kB 00:00 2024-03-21T13:03:54.047 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-21T13:03:54.072 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-21T13:03:54.114 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:56 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:03:54.239 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.2 MB/s | 548 kB 00:00 2024-03-21T13:03:54.265 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 750 k 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:54.266 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 750 k 2024-03-21T13:03:54.267 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 757 k 2024-03-21T13:03:54.267 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:03:54.281 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-03-21T13:03:54.314 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-03-21T13:03:54.407 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-03-21T13:03:54.466 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): ceph-mgr-dashboard-19.0.0-2267.g960fbbe 4.6 MB/s | 3.5 MB 00:00 2024-03-21T13:03:54.491 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 368 kB/s | 31 kB 00:00 2024-03-21T13:03:54.516 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 719 kB/s | 144 kB 00:00 2024-03-21T13:03:54.599 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-03-21T13:03:54.602 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:54.602 INFO:teuthology.orchestra.run.smithi089.stdout:Total 6.3 MB/s | 8.6 MB 00:01 2024-03-21T13:03:54.620 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:03:54.637 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:03:54.637 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:03:54.668 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-19.0.0-2267.g960fbbe4.el9.noarch.rpm 1.8 MB/s | 750 kB 00:00 2024-03-21T13:03:54.668 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:54.669 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.8 MB/s | 750 kB 00:00 2024-03-21T13:03:54.669 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:03:54.674 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:03:54.674 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:03:54.680 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:03:54.680 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:03:54.754 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:03:54.824 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:03:54.825 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:03:55.276 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:55.351 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:03:55.373 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-21T13:03:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-21T13:03:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-21T13:03:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-21T13:03:55.668 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-21T13:03:55.798 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-21T13:03:55.820 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:55.891 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-21T13:03:55.935 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-21T13:03:55.972 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:03:56.004 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 10/12 2024-03-21T13:03:56.119 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/1 2024-03-21T13:03:56.120 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:56.120 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:03:56.120 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:03:56.120 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:56.120 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:03:56.338 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-03-21T13:03:56.917 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:58 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:03:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 11/12 2024-03-21T13:03:57.071 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:03:57.071 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 145 k 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:03:57.072 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:57.073 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 145 k 2024-03-21T13:03:57.073 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 436 k 2024-03-21T13:03:57.073 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:03:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T13:03:57.273 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-19.0.0-2267.g960fbb 727 kB/s | 145 kB 00:00 2024-03-21T13:03:57.274 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:03:57.274 INFO:teuthology.orchestra.run.smithi152.stdout:Total 722 kB/s | 145 kB 00:00 2024-03-21T13:03:57.274 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:03:57.282 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:03:57.282 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:03:57.335 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:03:57.336 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:03:57.474 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:03:57.532 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T13:03:57.556 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T13:03:57.556 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:03:57.556 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T13:03:57.556 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 1/12 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 2/12 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 3/12 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-21T13:03:58.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-21T13:03:58.426 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-21T13:03:58.426 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:03:58.426 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-21T13:03:58.426 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-21T13:03:58.520 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4. 1/1 2024-03-21T13:03:58.520 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:58.520 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:03:58.520 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:03:58.520 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:03:58.520 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:03:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-21T13:03:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:58.769 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:03:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T13:03:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:03:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:58.770 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:03:58.821 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-03-21T13:03:59.046 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-21T13:03:59.398 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:01 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:03:59.496 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:03:59.548 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:03:59.549 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:03:59.549 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:03:59.619 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-03-21T13:03:59.620 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:13 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:03:59.780 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 7.4 M 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T13:03:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-03-21T13:03:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:03:59.783 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 37 M 2024-03-21T13:03:59.783 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 209 M 2024-03-21T13:03:59.783 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:00.036 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 215 kB/s | 32 kB 00:00 2024-03-21T13:04:00.069 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-21T13:04:00.111 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 843 kB/s | 34 kB 00:00 2024-03-21T13:04:00.161 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-21T13:04:00.183 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:02 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:00.328 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-21T13:04:00.334 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 3.5 M 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 34 k 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 16 k 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T13:04:00.336 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-03-21T13:04:00.337 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:00.338 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 8.6 M 2024-03-21T13:04:00.338 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 103 M 2024-03-21T13:04:00.338 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:00.425 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-21T13:04:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-21T13:04:00.596 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 803 kB 00:00 2024-03-21T13:04:00.755 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-21T13:04:00.897 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 655 kB/s | 195 kB 00:00 2024-03-21T13:04:00.997 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-21T13:04:01.051 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2267.g96 135 kB/s | 34 kB 00:00 2024-03-21T13:04:01.077 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2267.g960 60 kB/s | 16 kB 00:00 2024-03-21T13:04:01.130 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 6.0 MB/s | 7.4 MB 00:01 2024-03-21T13:04:01.133 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:01.133 INFO:teuthology.orchestra.run.smithi089.stdout:Total 27 MB/s | 37 MB 00:01 2024-03-21T13:04:01.172 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:01.194 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:01.194 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:01.218 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-21T13:04:01.252 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-21T13:04:01.485 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2267.g960fbbe 5.1 MB/s | 3.5 MB 00:00 2024-03-21T13:04:01.544 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-03-21T13:04:01.545 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:01.545 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:01.719 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-21T13:04:01.770 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 68 kB/s | 19 kB 00:00 2024-03-21T13:04:01.812 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.6 MB/s | 144 kB 00:00 2024-03-21T13:04:01.878 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-21T13:04:01.904 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-21T13:04:02.179 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-21T13:04:02.184 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:02.184 INFO:teuthology.orchestra.run.smithi152.stdout:Total 4.7 MB/s | 8.6 MB 00:01 2024-03-21T13:04:02.202 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:02.218 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:02.218 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:02.241 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:02.303 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-21T13:04:02.335 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-21T13:04:02.412 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:02.412 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:02.435 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:04:02.480 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-21T13:04:02.657 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:04:02.687 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:04:02.768 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-21T13:04:02.962 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:03.062 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-21T13:04:03.106 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-21T13:04:03.152 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-21T13:04:03.183 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-21T13:04:03.281 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-21T13:04:03.407 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-21T13:04:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-21T13:04:03.502 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-21T13:04:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-21T13:04:03.545 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-21T13:04:03.574 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:04:03.606 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 10/12 2024-03-21T13:04:04.657 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 11/12 2024-03-21T13:04:04.698 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T13:04:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-21T13:04:05.826 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-21T13:04:05.848 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 12/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el 1/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noa 2/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9 3/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-21T13:04:05.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T13:04:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:06.347 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:06.775 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-21T13:04:07.345 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:09 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:07.498 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 7.4 M 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T13:04:07.499 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-03-21T13:04:07.500 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:07.501 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 37 M 2024-03-21T13:04:07.501 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 209 M 2024-03-21T13:04:07.501 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:07.961 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 251 kB/s | 32 kB 00:00 2024-03-21T13:04:07.986 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 721 kB/s | 17 kB 00:00 2024-03-21T13:04:08.020 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-21T13:04:08.079 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-21T13:04:08.271 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-21T13:04:08.338 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-21T13:04:08.362 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-21T13:04:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T13:04:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 1/12 2024-03-21T13:04:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-21T13:04:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:04:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-21T13:04:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:04:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:04:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-21T13:04:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-21T13:04:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-21T13:04:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-21T13:04:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-21T13:04:08.655 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-03-21T13:04:08.963 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-21T13:04:08.963 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:08.963 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:08.964 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:09.395 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2024-03-21T13:04:09.455 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-22 4.6 MB/s | 7.4 MB 00:01 2024-03-21T13:04:09.563 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 215 kB/s | 195 kB 00:00 2024-03-21T13:04:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:23 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:10.114 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 459 kB/s | 803 kB 00:01 2024-03-21T13:04:10.115 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:10.116 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-21T13:04:10.116 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:04:10.116 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 49 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T13:04:10.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T13:04:10.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T13:04:10.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T13:04:10.118 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:10.118 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:10.118 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-21T13:04:10.118 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-03-21T13:04:10.118 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:10.119 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.0 M 2024-03-21T13:04:10.119 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2024-03-21T13:04:10.119 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:10.322 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 283 kB/s | 245 kB 00:00 2024-03-21T13:04:10.325 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:10.325 INFO:teuthology.orchestra.run.smithi152.stdout:Total 13 MB/s | 37 MB 00:02 2024-03-21T13:04:10.366 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:10.391 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:10.391 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:10.393 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 24 kB 00:00 2024-03-21T13:04:10.410 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-21T13:04:10.452 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-03-21T13:04:10.477 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-21T13:04:10.502 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-21T13:04:10.527 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-21T13:04:10.561 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): ceph-mgr-rook-19.0.0-2267.g960fbbe4.el9 167 kB/s | 49 kB 00:00 2024-03-21T13:04:10.753 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:10.753 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:10.786 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-21T13:04:10.812 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-03-21T13:04:10.962 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-21T13:04:10.987 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 476 kB/s | 203 kB 00:00 2024-03-21T13:04:11.096 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-03-21T13:04:11.130 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-21T13:04:11.133 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:11.134 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-21T13:04:11.134 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:11.147 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:11.147 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:11.238 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:11.238 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:11.520 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-21T13:04:11.568 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-21T13:04:11.663 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:04:11.696 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-21T13:04:11.742 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:11.877 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:04:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:04:11.910 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-21T13:04:11.963 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-21T13:04:11.980 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-21T13:04:12.020 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-21T13:04:12.057 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-21T13:04:12.099 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-21T13:04:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-21T13:04:12.301 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-21T13:04:12.359 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-21T13:04:12.654 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-21T13:04:12.758 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-21T13:04:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-21T13:04:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-21T13:04:13.149 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-21T13:04:13.198 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-21T13:04:13.218 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T13:04:14.019 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-21T13:04:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T13:04:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 1/13 2024-03-21T13:04:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-21T13:04:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-21T13:04:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-21T13:04:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-21T13:04:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-21T13:04:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-21T13:04:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-21T13:04:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-21T13:04:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-21T13:04:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-21T13:04:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-21T13:04:14.392 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-21T13:04:14.392 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:14.392 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:14.392 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:14.392 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T13:04:14.392 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:14.393 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:14.651 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2024-03-21T13:04:15.050 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-21T13:04:15.076 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T13:04:15.213 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:29 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:15.363 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 140 k 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T13:04:15.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T13:04:15.365 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T13:04:15.365 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:15.365 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:15.365 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-21T13:04:15.365 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-03-21T13:04:15.365 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:15.365 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 6.8 M 2024-03-21T13:04:15.366 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 33 M 2024-03-21T13:04:15.366 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:15.803 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-21T13:04:15.936 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-21T13:04:16.020 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2267.g960fbbe4.e 370 kB/s | 140 kB 00:00 2024-03-21T13:04:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 257 kB/s | 58 kB 00:00 2024-03-21T13:04:16.237 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 235 kB/s | 51 kB 00:00 2024-03-21T13:04:16.279 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-21T13:04:16.281 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:16.281 INFO:teuthology.orchestra.run.smithi089.stdout:Total 7.4 MB/s | 6.8 MB 00:00 2024-03-21T13:04:16.294 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:16.304 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:16.304 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:16.392 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:16.393 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:16.706 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:16.878 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-21T13:04:16.922 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-21T13:04:17.432 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-21T13:04:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-21T13:04:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-21T13:04:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 12/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fb 1/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-21T13:04:17.776 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-21T13:04:17.777 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-21T13:04:17.777 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-21T13:04:17.777 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-21T13:04:18.212 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-21T13:04:18.212 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:18.212 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:18.212 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:18.212 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T13:04:18.212 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:18.213 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:18.424 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T13:04:18.424 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/6 2024-03-21T13:04:18.424 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-21T13:04:18.424 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-21T13:04:18.424 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-21T13:04:18.425 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-21T13:04:18.631 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T13:04:18.716 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T13:04:18.717 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:18.717 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:18.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2024-03-21T13:04:19.201 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:21 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:19.351 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 49 k 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T13:04:19.352 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-03-21T13:04:19.353 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:19.354 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.0 M 2024-03-21T13:04:19.354 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-03-21T13:04:19.354 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:19.511 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:33 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:19.520 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 228 kB/s | 24 kB 00:00 2024-03-21T13:04:19.538 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 146 kB/s | 18 kB 00:00 2024-03-21T13:04:19.588 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-21T13:04:19.613 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-21T13:04:19.638 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-21T13:04:19.655 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-21T13:04:19.662 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:19.663 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:19.663 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:04:19.663 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:19.663 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 890 k 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-03-21T13:04:19.664 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:19.665 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 969 k 2024-03-21T13:04:19.665 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 2.8 M 2024-03-21T13:04:19.665 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:19.714 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): ceph-mgr-rook-19.0.0-2267.g960fbbe4.el9 164 kB/s | 49 kB 00:00 2024-03-21T13:04:19.731 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-03-21T13:04:19.747 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 163 kB/s | 15 kB 00:00 2024-03-21T13:04:19.799 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 587 kB/s | 79 kB 00:00 2024-03-21T13:04:19.806 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-21T13:04:19.881 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-03-21T13:04:19.906 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 894 kB/s | 90 kB 00:00 2024-03-21T13:04:19.940 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.9 MB/s | 1.0 MB 00:00 2024-03-21T13:04:19.941 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:19.942 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2024-03-21T13:04:19.942 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:19.955 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:19.955 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:20.045 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:20.045 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:20.124 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-fuse-19.0.0-2267.g960fbbe4.el9.x86_ 1.9 MB/s | 890 kB 00:00 2024-03-21T13:04:20.125 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:20.126 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.0 MB/s | 969 kB 00:00 2024-03-21T13:04:20.126 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:20.146 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:20.146 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:20.218 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:20.218 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:20.403 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:20.537 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-21T13:04:20.552 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:20.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T13:04:20.710 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-21T13:04:20.766 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-21T13:04:20.829 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-21T13:04:20.858 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-21T13:04:20.899 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-21T13:04:20.992 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-21T13:04:21.092 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-21T13:04:21.142 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-21T13:04:21.338 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T13:04:21.338 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/2 2024-03-21T13:04:21.650 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-21T13:04:21.650 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:21.650 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:21.650 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-21T13:04:21.650 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:21.650 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:21.859 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-21T13:04:21.932 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-21T13:04:21.939 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-volume 2024-03-21T13:04:21.965 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-21T13:04:22.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-21T13:04:22.035 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T13:04:22.513 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:36 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:22.667 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-21T13:04:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 273 k 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 342 k 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 1.5 M 2024-03-21T13:04:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 574 kB/s | 69 kB 00:00 2024-03-21T13:04:22.911 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 13/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 1/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-21T13:04:22.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-21T13:04:22.913 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-21T13:04:22.913 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-21T13:04:22.913 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-21T13:04:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-volume-19.0.0-2267.g960fbbe4.el9.no 1.1 MB/s | 273 kB 00:00 2024-03-21T13:04:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.3 MB/s | 342 kB 00:00 2024-03-21T13:04:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:22.933 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:22.933 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:22.965 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:22.965 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:23.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-21T13:04:23.267 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T13:04:23.268 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T13:04:23.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T13:04:23.269 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:23.269 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:23.293 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-21T13:04:23.326 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T13:04:23.342 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T13:04:23.342 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:04:23.342 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T13:04:23.342 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:23.497 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-03-21T13:04:24.066 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:26 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 1/2 2024-03-21T13:04:24.225 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 140 k 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T13:04:24.226 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T13:04:24.227 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T13:04:24.227 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T13:04:24.227 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:24.227 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:24.227 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-21T13:04:24.227 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-03-21T13:04:24.227 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:24.228 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 6.8 M 2024-03-21T13:04:24.228 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-03-21T13:04:24.228 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:24.511 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-21T13:04:24.511 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:24.511 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:24.512 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:24.512 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T13:04:24.512 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:24.512 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:24.740 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-21T13:04:24.765 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2267.g960fbbe4.e 470 kB/s | 140 kB 00:00 2024-03-21T13:04:24.799 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 748 kB/s | 248 kB 00:00 2024-03-21T13:04:24.831 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2024-03-21T13:04:25.058 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 197 kB/s | 58 kB 00:00 2024-03-21T13:04:25.083 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-03-21T13:04:25.192 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-21T13:04:25.194 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:25.195 INFO:teuthology.orchestra.run.smithi152.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-03-21T13:04:25.208 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:25.218 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:25.219 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:25.307 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:25.309 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:25.405 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:39 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:25.553 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 127 k 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:04:25.554 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:25.555 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 127 k 2024-03-21T13:04:25.555 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 456 k 2024-03-21T13:04:25.555 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:25.758 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-21T13:04:25.818 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-21T13:04:25.905 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-19.0.0-2267.g960fbbe4.el9.x86_64 362 kB/s | 127 kB 00:00 2024-03-21T13:04:25.906 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:25.906 INFO:teuthology.orchestra.run.smithi089.stdout:Total 360 kB/s | 127 kB 00:00 2024-03-21T13:04:25.906 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:25.912 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:25.912 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:26.184 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:26.329 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-21T13:04:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-21T13:04:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-21T13:04:26.538 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T13:04:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:27.063 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:27.064 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:27.064 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:27.064 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:27.064 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:27.064 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:27.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2024-03-21T13:04:27.340 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 6/6 2024-03-21T13:04:27.340 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 1/6 2024-03-21T13:04:27.340 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-21T13:04:27.340 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-21T13:04:27.341 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-21T13:04:27.341 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T13:04:27.649 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T13:04:27.650 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T13:04:27.650 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:27.650 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:27.860 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:41 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:27.887 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-03-21T13:04:27.959 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:28.013 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:28.014 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:04:28.014 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:28.072 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2024-03-21T13:04:28.450 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:30 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:28.600 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:28.600 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 890 k 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-03-21T13:04:28.601 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:28.602 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 969 k 2024-03-21T13:04:28.602 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 2.8 M 2024-03-21T13:04:28.602 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:28.636 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:42 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:28.743 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 560 kB/s | 79 kB 00:00 2024-03-21T13:04:28.784 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:28.785 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-21T13:04:28.785 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:28.785 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 31 k 2024-03-21T13:04:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:28.785 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:04:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 31 k 2024-03-21T13:04:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 140 k 2024-03-21T13:04:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:28.952 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-19.0.0-2267.g960fbbe4.el9.x86_6 188 kB/s | 31 kB 00:00 2024-03-21T13:04:28.953 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:28.953 INFO:teuthology.orchestra.run.smithi089.stdout:Total 187 kB/s | 31 kB 00:00 2024-03-21T13:04:28.953 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:28.956 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:28.956 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:28.982 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:28.983 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:29.009 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-fuse-19.0.0-2267.g960fbbe4.el9.x86_ 2.1 MB/s | 890 kB 00:00 2024-03-21T13:04:29.010 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:29.011 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.3 MB/s | 969 kB 00:00 2024-03-21T13:04:29.011 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:29.031 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:29.031 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:29.071 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:29.104 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:29.104 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:29.166 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:29.285 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:29.439 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-21T13:04:29.517 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T13:04:29.571 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:29.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:29.830 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:29.830 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:29.830 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:29.830 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:29.830 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:30.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2024-03-21T13:04:30.227 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2/2 2024-03-21T13:04:30.227 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/2 2024-03-21T13:04:30.534 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-21T13:04:30.535 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:30.535 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:30.535 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-21T13:04:30.535 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:30.535 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:30.564 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:44 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:30.662 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:30.715 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:30.716 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:04:30.716 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:30.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2024-03-21T13:04:30.855 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-volume 2024-03-21T13:04:31.385 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:45 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:31.425 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:33 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:31.487 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:31.542 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:31.543 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:04:31.543 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:31.581 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.0.0-2267.g960fbbe4.el9 ceph-noarch 273 k 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:31.582 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-03-21T13:04:31.583 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:31.583 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 342 k 2024-03-21T13:04:31.583 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 1.5 M 2024-03-21T13:04:31.583 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:31.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2024-03-21T13:04:31.705 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 566 kB/s | 69 kB 00:00 2024-03-21T13:04:31.955 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-volume-19.0.0-2267.g960fbbe4.el9.no 736 kB/s | 273 kB 00:00 2024-03-21T13:04:31.956 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:31.957 INFO:teuthology.orchestra.run.smithi152.stdout:Total 915 kB/s | 342 kB 00:00 2024-03-21T13:04:31.957 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:31.965 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:31.965 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:31.998 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:31.998 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:32.147 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:32.198 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:46 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:32.295 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-21T13:04:32.346 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:32.347 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:04:32.348 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T13:04:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2/2 2024-03-21T13:04:32.366 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:04:32.366 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T13:04:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:32.424 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2024-03-21T13:04:32.989 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:46 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:33.085 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:33.107 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 1/2 2024-03-21T13:04:33.136 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:33.137 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:04:33.137 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:33.206 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2024-03-21T13:04:33.439 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-21T13:04:33.439 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:33.439 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:33.439 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.0.0-2267.g960fbbe4.el9.noarch 2024-03-21T13:04:33.440 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T13:04:33.440 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:33.440 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:33.728 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-03-21T13:04:33.767 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:47 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:33.864 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:33.915 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:33.916 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:04:33.916 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:33.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2024-03-21T13:04:34.306 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:36 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:34.458 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:34.459 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:34.459 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-21T13:04:34.459 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 127 k 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 127 k 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 456 k 2024-03-21T13:04:34.460 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:34.554 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:48 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:34.651 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:34.677 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-19.0.0-2267.g960fbbe4.el9.x86_64 584 kB/s | 127 kB 00:00 2024-03-21T13:04:34.678 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:34.678 INFO:teuthology.orchestra.run.smithi152.stdout:Total 581 kB/s | 127 kB 00:00 2024-03-21T13:04:34.678 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:34.683 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:34.683 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:34.702 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:34.703 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-21T13:04:34.703 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:34.745 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:34.745 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:34.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2024-03-21T13:04:34.943 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:35.052 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:35.336 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:49 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:35.485 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 85 k 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:04:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:35.487 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 85 k 2024-03-21T13:04:35.487 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 231 k 2024-03-21T13:04:35.487 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:35.503 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:35.697 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 404 kB/s | 85 kB 00:00 2024-03-21T13:04:35.698 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:35.698 INFO:teuthology.orchestra.run.smithi089.stdout:Total 402 kB/s | 85 kB 00:00 2024-03-21T13:04:35.698 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:35.705 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:35.705 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:35.760 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:35.760 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:35.783 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:35.784 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:35.784 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:35.784 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:35.784 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:35.784 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:35.929 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:36.012 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-03-21T13:04:36.038 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:36.459 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:36.585 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:38 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:36.685 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:36.739 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:36.740 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:04:36.740 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:36.746 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:36.746 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:36.794 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-03-21T13:04:37.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-03-21T13:04:37.361 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:39 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:37.511 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:37.512 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:37.512 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-21T13:04:37.512 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:37.512 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:37.512 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 31 k 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 31 k 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 140 k 2024-03-21T13:04:37.513 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:37.597 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:51 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:37.750 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:37.750 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.1 M 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 13 M 2024-03-21T13:04:37.751 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:37.754 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-19.0.0-2267.g960fbbe4.el9.x86_6 130 kB/s | 31 kB 00:00 2024-03-21T13:04:37.755 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:37.755 INFO:teuthology.orchestra.run.smithi152.stdout:Total 129 kB/s | 31 kB 00:00 2024-03-21T13:04:37.755 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:37.759 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:37.759 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:37.784 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:37.784 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:37.882 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:37.994 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:38.352 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 5.1 MB/s | 3.1 MB 00:00 2024-03-21T13:04:38.353 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:38.353 INFO:teuthology.orchestra.run.smithi089.stdout:Total 5.1 MB/s | 3.1 MB 00:00 2024-03-21T13:04:38.353 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:38.363 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:38.363 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:38.437 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:38.438 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:38.636 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:38.636 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:38.637 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:38.637 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:38.637 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:38.637 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:38.818 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-03-21T13:04:38.869 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:38.945 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:38.969 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:38.969 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:04:38.969 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T13:04:38.969 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:04:38.969 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:04:38.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:39.382 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:41 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:39.482 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:39.534 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:39.535 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:04:39.535 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:39.593 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-03-21T13:04:40.029 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:40.030 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:40.030 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:40.030 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:40.030 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:40.030 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:40.161 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:42 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:40.259 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:40.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2024-03-21T13:04:40.311 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:40.311 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:04:40.311 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:40.370 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-03-21T13:04:40.860 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:54 ago on Thu 21 Mar 2024 01:02:46 PM UTC. 2024-03-21T13:04:40.942 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:42 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:41.007 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-21T13:04:41.008 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-03-21T13:04:41.008 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:41.008 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-21T13:04:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 170 k 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 170 k 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 490 k 2024-03-21T13:04:41.009 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-21T13:04:41.040 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:41.091 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:41.092 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:04:41.093 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:41.163 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-03-21T13:04:41.324 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 542 kB/s | 170 kB 00:00 2024-03-21T13:04:41.324 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:41.324 INFO:teuthology.orchestra.run.smithi089.stdout:Total 540 kB/s | 170 kB 00:00 2024-03-21T13:04:41.325 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-21T13:04:41.332 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-21T13:04:41.332 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-21T13:04:41.388 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-21T13:04:41.388 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-21T13:04:41.575 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-21T13:04:41.672 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:41.734 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:43 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:41.833 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:41.887 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:41.888 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:04:41.888 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:41.954 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-03-21T13:04:42.185 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:42.478 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:42.478 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:42.478 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-21T13:04:42.478 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:42.478 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:04:42.478 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-21T13:04:42.518 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:44 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:42.615 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:42.666 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:42.667 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:04:42.667 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:42.729 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-03-21T13:04:42.744 DEBUG:teuthology.parallel:result is None 2024-03-21T13:04:43.294 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:45 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:43.391 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 is already installed. 2024-03-21T13:04:43.442 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:43.443 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-21T13:04:43.443 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:43.500 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-03-21T13:04:44.061 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:46 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:44.212 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 85 k 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 85 k 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 231 k 2024-03-21T13:04:44.213 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:44.540 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 261 kB/s | 85 kB 00:00 2024-03-21T13:04:44.541 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:44.541 INFO:teuthology.orchestra.run.smithi152.stdout:Total 260 kB/s | 85 kB 00:00 2024-03-21T13:04:44.541 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:44.548 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:44.549 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:44.606 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:44.606 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:44.849 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:45.251 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:45.575 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:45.575 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:45.575 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:45.575 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:45.575 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:45.575 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:45.828 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-03-21T13:04:46.428 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:48 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 3.1 M 2024-03-21T13:04:46.586 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:46.587 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:46.587 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:46.587 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:04:46.587 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:46.587 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.1 M 2024-03-21T13:04:46.587 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 13 M 2024-03-21T13:04:46.587 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:47.977 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 2.2 MB/s | 3.1 MB 00:01 2024-03-21T13:04:47.977 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:47.978 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.2 MB/s | 3.1 MB 00:01 2024-03-21T13:04:47.978 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:47.989 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:47.989 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:48.066 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:48.066 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:48.490 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:48.552 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:48.577 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:48.577 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:04:48.577 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T13:04:48.577 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:04:48.577 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:04:48.578 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:49.742 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:49.742 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:49.743 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:49.743 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:49.743 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:49.743 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:50.000 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-03-21T13:04:50.577 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:52 ago on Thu 21 Mar 2024 01:02:58 PM UTC. 2024-03-21T13:04:50.727 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.0.0-2267.g960fbbe4.el9 ceph 170 k 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-21T13:04:50.728 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-21T13:04:50.729 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-21T13:04:50.729 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:50.729 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 170 k 2024-03-21T13:04:50.729 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 490 k 2024-03-21T13:04:50.729 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-21T13:04:51.437 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-19.0.0-2267.g960fbbe4.el9.x86_64.rpm 241 kB/s | 170 kB 00:00 2024-03-21T13:04:51.438 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:04:51.438 INFO:teuthology.orchestra.run.smithi152.stdout:Total 240 kB/s | 170 kB 00:00 2024-03-21T13:04:51.438 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-21T13:04:51.447 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-21T13:04:51.447 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-21T13:04:51.506 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-21T13:04:51.506 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-21T13:04:51.653 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-21T13:04:51.753 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:52.200 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:52.520 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 1/1 2024-03-21T13:04:52.520 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:52.520 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-21T13:04:52.520 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.0.0-2267.g960fbbe4.el9.x86_64 2024-03-21T13:04:52.520 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:04:52.520 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-21T13:04:52.767 DEBUG:teuthology.parallel:result is None 2024-03-21T13:04:52.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:53.121 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T13:04:53.146 INFO:teuthology.orchestra.run.smithi089.stdout:19.0.0-2267.g960fbbe4.el9 2024-03-21T13:04:53.146 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2267.g960fbbe4.el9 2024-03-21T13:04:53.146 INFO:teuthology.task.install:The correct ceph version 19.0.0-2267.g960fbbe4 is installed. 2024-03-21T13:04:53.148 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:53.471 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T13:04:53.493 INFO:teuthology.orchestra.run.smithi152.stdout:19.0.0-2267.g960fbbe4.el9 2024-03-21T13:04:53.493 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2267.g960fbbe4.el9 2024-03-21T13:04:53.493 INFO:teuthology.task.install:The correct ceph version 19.0.0-2267.g960fbbe4 is installed. 2024-03-21T13:04:53.495 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-21T13:04:53.495 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:04:53.495 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T13:04:53.527 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:04:53.527 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T13:04:53.562 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-21T13:04:53.562 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:04:53.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T13:04:53.597 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T13:04:53.666 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:04:53.666 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T13:04:53.694 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T13:04:53.763 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-21T13:04:53.763 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:04:53.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T13:04:53.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T13:04:53.861 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:04:53.861 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T13:04:53.889 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T13:04:53.957 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-21T13:04:53.957 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:04:53.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T13:04:53.987 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T13:04:54.055 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:04:54.055 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T13:04:54.083 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T13:04:54.150 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-21T13:04:54.253 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'cephadm_mode': 'root'} 2024-03-21T13:04:54.253 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:54.253 INFO:tasks.cephadm:Cluster fsid is 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:04:54.254 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-21T13:04:54.254 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.89', 'mon.c': '[v2:172.21.15.89:3301,v1:172.21.15.89:6790]', 'mon.b': '172.21.15.152'} 2024-03-21T13:04:54.254 INFO:tasks.cephadm:First mon is mon.a on smithi089 2024-03-21T13:04:54.254 INFO:tasks.cephadm:First mgr is y 2024-03-21T13:04:54.254 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-21T13:04:54.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2024-03-21T13:04:54.284 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-03-21T13:04:54.312 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-21T13:04:54.312 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:54.534 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '960fbbe4645c2becc8084669be194c95183c7753', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77830/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2267-g960fbbe4', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2267.g960fbbe4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-21 09:55:14.969817', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-2024-03-21-0818', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-21T13:04:54.921 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-testing-2024-03-21-0818, sha1 960fbbe4645c2becc8084669be194c95183c7753 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:54.923 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/x86_64/flavors/default/cephadm 2024-03-21T13:04:54.923 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T13:04:56.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r--. 1 ubuntu ubuntu 769397 Mar 21 13:04 /home/ubuntu/cephtest/cephadm 2024-03-21T13:04:56.292 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-21-0818/960fbbe4645c2becc8084669be194c95183c7753/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T13:04:57.624 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-r--r--. 1 ubuntu ubuntu 769397 Mar 21 13:04 /home/ubuntu/cephtest/cephadm 2024-03-21T13:04:57.625 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T13:04:57.645 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T13:04:57.675 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 on all hosts... 2024-03-21T13:04:57.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 pull 2024-03-21T13:04:57.687 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 pull 2024-03-21T13:04:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T13:04:58.630 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T13:05:26.161 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-21T13:05:26.161 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)", 2024-03-21T13:05:26.161 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e", 2024-03-21T13:05:26.161 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-03-21T13:05:26.161 INFO:teuthology.orchestra.run.smithi152.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93" 2024-03-21T13:05:26.162 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-03-21T13:05:26.162 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-21T13:05:31.688 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-21T13:05:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)", 2024-03-21T13:05:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "image_id": "19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e", 2024-03-21T13:05:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "repo_digests": [ 2024-03-21T13:05:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93" 2024-03-21T13:05:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-03-21T13:05:31.689 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-21T13:05:31.713 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2024-03-21T13:05:31.757 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-03-21T13:05:31.793 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2024-03-21T13:05:31.824 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-03-21T13:05:31.865 INFO:tasks.cephadm:Writing seed config... 2024-03-21T13:05:31.866 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-21T13:05:31.866 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-21T13:05:31.867 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-21T13:05:31.868 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-21T13:05:31.868 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:05:31.868 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-21T13:05:31.883 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9c355834-e783-11ee-95cd-87774f69a715 mon election default strategy = 3 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-21T13:05:31.883 DEBUG:teuthology.orchestra.run.smithi089:mon.a> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a.service 2024-03-21T13:05:31.925 DEBUG:teuthology.orchestra.run.smithi089:mgr.y> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y.service 2024-03-21T13:05:31.968 INFO:tasks.cephadm:Bootstrapping... 2024-03-21T13:05:31.968 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 -v bootstrap --fsid 9c355834-e783-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.89 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:05:32.159 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:05:32.159 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753', '-v', 'bootstrap', '--fsid', '9c355834-e783-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.89', '--skip-admin-label'] 2024-03-21T13:05:32.159 INFO:teuthology.orchestra.run.smithi089.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-21T13:05:32.160 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying podman|docker is present... 2024-03-21T13:05:32.183 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T13:05:32.184 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying lvm2 is present... 2024-03-21T13:05:32.184 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying time synchronization is in place... 2024-03-21T13:05:32.191 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T13:05:32.191 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T13:05:32.197 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T13:05:32.197 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-03-21T13:05:32.203 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2024-03-21T13:05:32.209 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2024-03-21T13:05:32.209 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2024-03-21T13:05:32.209 INFO:teuthology.orchestra.run.smithi089.stdout:Repeating the final host check... 2024-03-21T13:05:32.232 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T13:05:32.232 INFO:teuthology.orchestra.run.smithi089.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-21T13:05:32.232 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl is present 2024-03-21T13:05:32.232 INFO:teuthology.orchestra.run.smithi089.stdout:lvcreate is present 2024-03-21T13:05:32.238 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T13:05:32.238 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T13:05:32.244 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T13:05:32.244 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-03-21T13:05:32.250 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2024-03-21T13:05:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2024-03-21T13:05:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2024-03-21T13:05:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:Host looks OK 2024-03-21T13:05:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:Cluster fsid: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:05:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:Acquiring lock 139671701480400 on /run/cephadm/9c355834-e783-11ee-95cd-87774f69a715.lock 2024-03-21T13:05:32.258 INFO:teuthology.orchestra.run.smithi089.stdout:Lock 139671701480400 acquired on /run/cephadm/9c355834-e783-11ee-95cd-87774f69a715.lock 2024-03-21T13:05:32.258 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 3300 ... 2024-03-21T13:05:32.259 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 6789 ... 2024-03-21T13:05:32.259 INFO:teuthology.orchestra.run.smithi089.stdout:Base mon IP(s) is [172.21.15.89:3300, 172.21.15.89:6789], mon addrv is [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-03-21T13:05:32.260 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.89 metric 100 2024-03-21T13:05:32.261 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.89 metric 100 2024-03-21T13:05:32.263 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-21T13:05:32.263 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-21T13:05:32.264 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-03-21T13:05:32.266 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-21T13:05:32.266 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-21T13:05:32.266 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T13:05:32.266 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-21T13:05:32.266 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-03-21T13:05:32.266 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T13:05:32.267 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2024-03-21T13:05:32.267 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2024-03-21T13:05:32.267 INFO:teuthology.orchestra.run.smithi089.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-21T13:05:32.267 INFO:teuthology.orchestra.run.smithi089.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-21T13:05:32.267 INFO:teuthology.orchestra.run.smithi089.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T13:05:32.718 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e 2024-03-21T13:05:32.718 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753... 2024-03-21T13:05:32.718 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Getting image source signatures 2024-03-21T13:05:32.718 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:42ed92a376f0fc24f900533b2c051b7c28b467ca85a1aa456c3a11e39d805f31 2024-03-21T13:05:32.718 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-21T13:05:32.718 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying config sha256:19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e 2024-03-21T13:05:32.718 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-21T13:05:33.664 INFO:teuthology.orchestra.run.smithi089.stdout:ceph: stdout ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev) 2024-03-21T13:05:33.664 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph version: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev) 2024-03-21T13:05:33.664 INFO:teuthology.orchestra.run.smithi089.stdout:Extracting ceph user uid/gid from container image... 2024-03-21T13:05:34.727 INFO:teuthology.orchestra.run.smithi089.stdout:stat: stdout 167 167 2024-03-21T13:05:34.728 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial keys... 2024-03-21T13:05:35.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQCfMPxlgoTBBhAAp5r2yE4rCD7QHbIuzXnZRw== 2024-03-21T13:05:36.631 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQCfMPxlH3pIOBAA/NyQG8CNxg030cQ8u0eH6g== 2024-03-21T13:05:37.594 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQCgMPxlM5QdOhAAeK4T0f+r3qo0bqHMJ6eUqQ== 2024-03-21T13:05:37.595 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial monmap... 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool for a [v2:172.21.15.89:3300,v1:172.21.15.89:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = quincy 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: set fsid to 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:05:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mon... 2024-03-21T13:05:39.523 INFO:teuthology.orchestra.run.smithi089.stdout:create mon.a on 2024-03-21T13:05:39.718 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-21T13:05:39.857 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-21T13:05:40.027 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9c355834-e783-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-9c355834-e783-11ee-95cd-87774f69a715.target. 2024-03-21T13:05:40.027 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9c355834-e783-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-9c355834-e783-11ee-95cd-87774f69a715.target. 2024-03-21T13:05:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a 2024-03-21T13:05:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-21T13:05:40.413 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9c355834-e783-11ee-95cd-87774f69a715.target.wants/ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-9c355834-e783-11ee-95cd-87774f69a715@.service. 2024-03-21T13:05:41.039 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:05:41.039 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-03-21T13:05:41.046 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:05:41.046 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-03-21T13:05:41.046 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-03-21T13:05:41.046 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T13:05:41.046 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon to start... 2024-03-21T13:05:41.046 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon... 2024-03-21T13:05:41.992 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout cluster: 2024-03-21T13:05:41.998 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout id: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:05:41.998 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-21T13:05:41.998 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:41.998 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout services: 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558188s) 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout data: 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pgs: 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.455+0000 7f405e38f700 1 Processor -- start 2024-03-21T13:05:41.999 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.455+0000 7f405e38f700 1 -- start start 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.455+0000 7f405e38f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f40581062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.455+0000 7f405e38f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581067f0 con 0x7f4058105e90 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.455+0000 7f4057fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f40581062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.455+0000 7f4057fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f40581062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51070/0 (socket says 172.21.15.89:51070) 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.455+0000 7f4057fff700 1 -- 172.21.15.89:0/1120205851 learned_addr learned my addr 172.21.15.89:0/1120205851 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.456+0000 7f4057fff700 1 -- 172.21.15.89:0/1120205851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058107080 con 0x7f4058105e90 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.456+0000 7f4057fff700 1 --2- 172.21.15.89:0/1120205851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f40581062b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f40480098a0 tx=0x7f4048009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee61584338519305 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.456+0000 7f4056ffd700 1 -- 172.21.15.89:0/1120205851 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4048004160 con 0x7f4058105e90 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.456+0000 7f4056ffd700 1 -- 172.21.15.89:0/1120205851 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048004300 con 0x7f4058105e90 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.456+0000 7f4056ffd700 1 -- 172.21.15.89:0/1120205851 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40480036a0 con 0x7f4058105e90 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.456+0000 7f405e38f700 1 -- 172.21.15.89:0/1120205851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 msgr2=0x7f40581062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.456+0000 7f405e38f700 1 --2- 172.21.15.89:0/1120205851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f40581062b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f40480098a0 tx=0x7f4048009bb0 comp rx=0 tx=0).stop 2024-03-21T13:05:42.000 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 -- 172.21.15.89:0/1120205851 shutdown_connections 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 --2- 172.21.15.89:0/1120205851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f40581062b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 -- 172.21.15.89:0/1120205851 >> 172.21.15.89:0/1120205851 conn(0x7f4058101480 msgr2=0x7f40581038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 -- 172.21.15.89:0/1120205851 shutdown_connections 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 -- 172.21.15.89:0/1120205851 wait complete. 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 Processor -- start 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 -- start start 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f405807b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.457+0000 7f405e38f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4058107550 con 0x7f4058105e90 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f4057fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f405807b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f4057fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f405807b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51074/0 (socket says 172.21.15.89:51074) 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f4057fff700 1 -- 172.21.15.89:0/1509252407 learned_addr learned my addr 172.21.15.89:0/1509252407 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f4057fff700 1 -- 172.21.15.89:0/1509252407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405807b8c0 con 0x7f4058105e90 2024-03-21T13:05:42.001 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f4057fff700 1 --2- 172.21.15.89:0/1509252407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f405807b380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4048000c00 tx=0x7f4048004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f40557fa700 1 -- 172.21.15.89:0/1509252407 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4048003db0 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f40557fa700 1 -- 172.21.15.89:0/1509252407 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048011630 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4058079ab0 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f40557fa700 1 -- 172.21.15.89:0/1509252407 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4048011950 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4058079ff0 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f40557fa700 1 -- 172.21.15.89:0/1509252407 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f404801a460 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.458+0000 7f40557fa700 1 -- 172.21.15.89:0/1509252407 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f404801b780 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.459+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40580637c0 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.461+0000 7f40557fa700 1 -- 172.21.15.89:0/1509252407 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4048031020 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.530+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4058192b90 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.530+0000 7f40557fa700 1 -- 172.21.15.89:0/1509252407 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f404801b9d0 con 0x7f4058105e90 2024-03-21T13:05:42.002 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.531+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 msgr2=0x7f405807b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.531+0000 7f405e38f700 1 --2- 172.21.15.89:0/1509252407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f405807b380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4048000c00 tx=0x7f4048004750 comp rx=0 tx=0).stop 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.531+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 shutdown_connections 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.531+0000 7f405e38f700 1 --2- 172.21.15.89:0/1509252407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4058105e90 0x7f405807b380 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.531+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 >> 172.21.15.89:0/1509252407 conn(0x7f4058101480 msgr2=0x7f4058190180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.531+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 shutdown_connections 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:41.531+0000 7f405e38f700 1 -- 172.21.15.89:0/1509252407 wait complete. 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:mon is available 2024-03-21T13:05:42.003 INFO:teuthology.orchestra.run.smithi089.stdout:Assimilating anything we can from ceph.conf... 2024-03-21T13:05:43.063 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:42 smithi089 ceph-mon[29611]: from='client.? 172.21.15.89:0/679438143' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T13:05:43.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:43.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T13:05:43.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:05:43.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-03-21T13:05:43.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.409+0000 7f9fe73aa700 1 Processor -- start 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe73aa700 1 -- start start 2024-03-21T13:05:43.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe73aa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe73aa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01067f0 con 0x7f9fe0105e90 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe5146700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe5146700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51084/0 (socket says 172.21.15.89:51084) 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe5146700 1 -- 172.21.15.89:0/1579627034 learned_addr learned my addr 172.21.15.89:0/1579627034 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe5146700 1 -- 172.21.15.89:0/1579627034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0107080 con 0x7f9fe0105e90 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.410+0000 7f9fe5146700 1 --2- 172.21.15.89:0/1579627034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01062b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00098a0 tx=0x7f9fd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d217c225633ab755 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fd7fff700 1 -- 172.21.15.89:0/1579627034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fd0004160 con 0x7f9fe0105e90 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fd7fff700 1 -- 172.21.15.89:0/1579627034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd0004300 con 0x7f9fe0105e90 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fd7fff700 1 -- 172.21.15.89:0/1579627034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fd00036a0 con 0x7f9fe0105e90 2024-03-21T13:05:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fe73aa700 1 -- 172.21.15.89:0/1579627034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 msgr2=0x7f9fe01062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fe73aa700 1 --2- 172.21.15.89:0/1579627034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01062b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00098a0 tx=0x7f9fd0009bb0 comp rx=0 tx=0).stop 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fe73aa700 1 -- 172.21.15.89:0/1579627034 shutdown_connections 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fe73aa700 1 --2- 172.21.15.89:0/1579627034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01062b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fe73aa700 1 -- 172.21.15.89:0/1579627034 >> 172.21.15.89:0/1579627034 conn(0x7f9fe0101480 msgr2=0x7f9fe01038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fe73aa700 1 -- 172.21.15.89:0/1579627034 shutdown_connections 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.411+0000 7f9fe73aa700 1 -- 172.21.15.89:0/1579627034 wait complete. 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe73aa700 1 Processor -- start 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe73aa700 1 -- start start 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe73aa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe73aa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0107550 con 0x7f9fe0105e90 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe5146700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:43.075 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe5146700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51088/0 (socket says 172.21.15.89:51088) 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe5146700 1 -- 172.21.15.89:0/679438143 learned_addr learned my addr 172.21.15.89:0/679438143 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.412+0000 7f9fe5146700 1 -- 172.21.15.89:0/679438143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0199b20 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fe5146700 1 --2- 172.21.15.89:0/679438143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01995e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0000c00 tx=0x7f9fd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fd0003db0 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd0011630 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe0199db0 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fd0011950 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe019caa0 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9fd001a460 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.413+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9fd001b780 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.414+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe00637c0 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.416+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9fd0031020 con 0x7f9fe0105e90 2024-03-21T13:05:43.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.484+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f9fe0192070 con 0x7f9fe0105e90 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.486+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+338 (secure 0 0 0) 0x7f9fd001b9d0 con 0x7f9fe0105e90 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.487+0000 7f9fd67fc700 1 -- 172.21.15.89:0/679438143 <== mon.0 v2:172.21.15.89:3300/0 8 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f9fd001ad40 con 0x7f9fe0105e90 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.488+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 msgr2=0x7f9fe01995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.488+0000 7f9fe73aa700 1 --2- 172.21.15.89:0/679438143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01995e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0000c00 tx=0x7f9fd0004750 comp rx=0 tx=0).stop 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.488+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 shutdown_connections 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.488+0000 7f9fe73aa700 1 --2- 172.21.15.89:0/679438143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0105e90 0x7f9fe01995e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.488+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 >> 172.21.15.89:0/679438143 conn(0x7f9fe0101480 msgr2=0x7f9fe0190180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.488+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 shutdown_connections 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:42.488+0000 7f9fe73aa700 1 -- 172.21.15.89:0/679438143 wait complete. 2024-03-21T13:05:43.077 INFO:teuthology.orchestra.run.smithi089.stdout:Generating new minimal ceph.conf... 2024-03-21T13:05:44.235 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:43 smithi089 ceph-mon[29611]: from='client.? 172.21.15.89:0/2660591059' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:44.242 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.524+0000 7fe0c3240700 1 Processor -- start 2024-03-21T13:05:44.242 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.524+0000 7fe0c3240700 1 -- start start 2024-03-21T13:05:44.242 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0c3240700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc1062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0c3240700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc1067f0 con 0x7fe0bc105e90 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0c0fdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc1062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0c0fdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc1062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51094/0 (socket says 172.21.15.89:51094) 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0c0fdc700 1 -- 172.21.15.89:0/3776562807 learned_addr learned my addr 172.21.15.89:0/3776562807 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0c0fdc700 1 -- 172.21.15.89:0/3776562807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc107080 con 0x7fe0bc105e90 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0c0fdc700 1 --2- 172.21.15.89:0/3776562807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc1062b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00098a0 tx=0x7fe0b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ef24c002f02bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0bb7fe700 1 -- 172.21.15.89:0/3776562807 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0b0004160 con 0x7fe0bc105e90 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0bb7fe700 1 -- 172.21.15.89:0/3776562807 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fe0b0004300 con 0x7fe0bc105e90 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.525+0000 7fe0bb7fe700 1 -- 172.21.15.89:0/3776562807 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0b0011940 con 0x7fe0bc105e90 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 -- 172.21.15.89:0/3776562807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 msgr2=0x7fe0bc1062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 --2- 172.21.15.89:0/3776562807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc1062b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00098a0 tx=0x7fe0b0009bb0 comp rx=0 tx=0).stop 2024-03-21T13:05:44.243 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 -- 172.21.15.89:0/3776562807 shutdown_connections 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 --2- 172.21.15.89:0/3776562807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc1062b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 -- 172.21.15.89:0/3776562807 >> 172.21.15.89:0/3776562807 conn(0x7fe0bc101480 msgr2=0x7fe0bc1038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 -- 172.21.15.89:0/3776562807 shutdown_connections 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 -- 172.21.15.89:0/3776562807 wait complete. 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.526+0000 7fe0c3240700 1 Processor -- start 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c3240700 1 -- start start 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c3240700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c3240700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc106970 con 0x7fe0bc105e90 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c0fdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c0fdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51098/0 (socket says 172.21.15.89:51098) 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c0fdc700 1 -- 172.21.15.89:0/2660591059 learned_addr learned my addr 172.21.15.89:0/2660591059 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c0fdc700 1 -- 172.21.15.89:0/2660591059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc19a290 con 0x7fe0bc105e90 2024-03-21T13:05:44.244 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c0fdc700 1 --2- 172.21.15.89:0/2660591059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc199d50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009fb0 tx=0x7fe0b00119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0b9ffb700 1 -- 172.21.15.89:0/2660591059 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0b001a430 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0b9ffb700 1 -- 172.21.15.89:0/2660591059 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fe0b001aa30 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0bc19a520 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.527+0000 7fe0b9ffb700 1 -- 172.21.15.89:0/2660591059 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0b0024860 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.528+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0bc19d210 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.528+0000 7fe0b9ffb700 1 -- 172.21.15.89:0/2660591059 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe0b0022020 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.528+0000 7fe0b9ffb700 1 -- 172.21.15.89:0/2660591059 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe0b00193c0 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.528+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0bc0637c0 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.531+0000 7fe0b9ffb700 1 -- 172.21.15.89:0/2660591059 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe0b002ddf0 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.598+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe0bc19d5f0 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.598+0000 7fe0b9ffb700 1 -- 172.21.15.89:0/2660591059 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fe0b0032030 con 0x7fe0bc105e90 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.599+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 msgr2=0x7fe0bc199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.599+0000 7fe0c3240700 1 --2- 172.21.15.89:0/2660591059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc199d50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009fb0 tx=0x7fe0b00119c0 comp rx=0 tx=0).stop 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.599+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 shutdown_connections 2024-03-21T13:05:44.245 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.599+0000 7fe0c3240700 1 --2- 172.21.15.89:0/2660591059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0bc105e90 0x7fe0bc199d50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:44.246 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.599+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 >> 172.21.15.89:0/2660591059 conn(0x7fe0bc101480 msgr2=0x7fe0bc101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:44.246 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.599+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 shutdown_connections 2024-03-21T13:05:44.246 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:43.599+0000 7fe0c3240700 1 -- 172.21.15.89:0/2660591059 wait complete. 2024-03-21T13:05:44.246 INFO:teuthology.orchestra.run.smithi089.stdout:Restarting the monitor... 2024-03-21T13:05:44.501 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:44 smithi089 systemd[1]: Stopping Ceph mon.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:05:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:44 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a[29588]: 2024-03-21T13:05:44.499+0000 7f2dd9ae7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:05:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:44 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a[29588]: 2024-03-21T13:05:44.499+0000 7f2dd9ae7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-21T13:05:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:44 smithi089 podman[29936]: 2024-03-21 13:05:44.70513054 +0000 UTC m=+0.312004735 container died d8ff70e2bccf9ac8e7bd96743794cce09e387cdc7cc99200a5fe030307d580a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319) 2024-03-21T13:05:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 podman[29936]: 2024-03-21 13:05:45.072155707 +0000 UTC m=+0.679029942 container cleanup d8ff70e2bccf9ac8e7bd96743794cce09e387cdc7cc99200a5fe030307d580a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-21T13:05:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 bash[29936]: ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a 2024-03-21T13:05:45.690 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 podman[29955]: 2024-03-21 13:05:45.331118912 +0000 UTC m=+0.628265065 container remove d8ff70e2bccf9ac8e7bd96743794cce09e387cdc7cc99200a5fe030307d580a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T13:05:45.690 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T13:05:45.690 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 systemd[1]: Stopped Ceph mon.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:05:45.690 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 systemd[1]: Starting Ceph mon.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:05:45.929 INFO:teuthology.orchestra.run.smithi089.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-21T13:05:45.966 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 podman[30057]: 2024-03-21 13:05:45.690306567 +0000 UTC m=+0.104325169 container create 7c27706dcbfb3d96e18d3a1220b981f232d12ddc40388c94ba5f816d8cae4954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-21T13:05:45.966 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 podman[30057]: 2024-03-21 13:05:45.603630711 +0000 UTC m=+0.017649319 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:45.966 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 podman[30057]: 2024-03-21 13:05:45.859261476 +0000 UTC m=+0.273280081 container init 7c27706dcbfb3d96e18d3a1220b981f232d12ddc40388c94ba5f816d8cae4954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 podman[30057]: 2024-03-21 13:05:45.865077888 +0000 UTC m=+0.279096490 container start 7c27706dcbfb3d96e18d3a1220b981f232d12ddc40388c94ba5f816d8cae4954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 7 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: pidfile_write: ignore empty --pid-file 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: load: jerasure load: lrc 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: RocksDB version: 7.9.2 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Git sha 0 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: DB SUMMARY 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: DB Session ID: 7R14PWHEDFEJ938P87Q5 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: CURRENT file: CURRENT 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T13:05:45.967 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-21T13:05:45.968 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-21T13:05:45.968 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 87145 ; 2024-03-21T13:05:45.969 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.error_if_exists: 0 2024-03-21T13:05:45.969 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.create_if_missing: 0 2024-03-21T13:05:45.969 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.paranoid_checks: 1 2024-03-21T13:05:45.969 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T13:05:45.969 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.env: 0x555e80564b20 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.info_log: 0x555e8293f920 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.statistics: (nil) 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.use_fsync: 0 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_log_file_size: 0 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T13:05:45.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.allow_fallocate: 1 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.use_direct_reads: 0 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.db_log_dir: 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.wal_dir: 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T13:05:45.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.write_buffer_manager: 0x555e829101e0 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.unordered_write: 0 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T13:05:45.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.row_cache: None 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.wal_filter: None 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.two_write_queues: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.wal_compression: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.atomic_flush: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.log_readahead_size: 0 2024-03-21T13:05:45.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_background_jobs: 2 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_background_compactions: -1 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_subcompactions: 1 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T13:05:45.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_open_files: -1 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_background_flushes: -1 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Compression algorithms supported: 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kZSTD supported: 0 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kXpressCompression supported: 0 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T13:05:45.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kZlibCompression supported: 1 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kSnappyCompression supported: 1 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kLZ4Compression supported: 1 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: kBZip2Compression supported: 0 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.merge_operator: 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_filter: None 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T13:05:45.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555e8293fa20) 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: cache_index_and_filter_blocks: 1 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: pin_top_level_index_and_filter: 1 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_type: 0 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: data_block_index_type: 0 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_shortening: 1 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: checksum: 4 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: no_block_cache: 0 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache: 0x555e82943090 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_name: BinnedLRUCache 2024-03-21T13:05:45.977 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_options: 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: capacity : 536870912 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: num_shard_bits : 4 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: strict_capacity_limit : 0 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: high_pri_pool_ratio: 0.000 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_compressed: (nil) 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: persistent_cache: (nil) 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_size: 4096 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_size_deviation: 10 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_restart_interval: 16 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_block_restart_interval: 1 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: metadata_block_size: 4096 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: partition_filters: 0 2024-03-21T13:05:45.978 INFO:journalctl@ceph.mon.a.smithi089.stdout: use_delta_encoding: 1 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: filter_policy: bloomfilter 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: whole_key_filtering: 1 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: verify_compression: 0 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: read_amp_bytes_per_bit: 0 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: format_version: 5 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: enable_index_compression: 1 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_align: 0 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: max_auto_readahead_size: 262144 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: prepopulate_block_cache: 0 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: initial_auto_readahead_size: 8192 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression: NoCompression 2024-03-21T13:05:45.979 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.num_levels: 7 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T13:05:45.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T13:05:45.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T13:05:45.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T13:05:45.983 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T13:05:45.983 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T13:05:45.983 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T13:05:45.983 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.table_properties_collectors: 2024-03-21T13:05:45.986 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.inplace_update_support: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.bloom_locality: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.max_successive_merges: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.ttl: 2592000 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T13:05:45.987 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.enable_blob_files: false 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.min_blob_size: 0 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2f73474d-2863-4ade-96e9-0106026f29c5 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026345893437, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-21T13:05:45.988 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026345894354, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 83567, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 262, "table_properties": {"data_size": 81739, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10592, "raw_average_key_size": 47, "raw_value_size": 75751, "raw_average_value_size": 342, "num_data_blocks": 10, "num_entries": 221, "num_filter_entries": 221, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711026345, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2f73474d-2863-4ade-96e9-0106026f29c5", "db_session_id": "7R14PWHEDFEJ938P87Q5", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026345894437, "job": 1, "event": "recovery_finished"} 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555e82a32000 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: DB pointer 0x555e82a1c000 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** DB Stats ** 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:05:45.989 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** Compaction Stats [default] ** 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: L0 2/0 83.44 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: Sum 2/0 83.44 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** Compaction Stats [default] ** 2024-03-21T13:05:45.990 INFO:journalctl@ceph.mon.a.smithi089.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative compaction: 0.00 GB write, 28.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval compaction: 0.00 GB write, 28.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Block cache BinnedLRUCache@0x555e82943090#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-21T13:05:45.991 INFO:journalctl@ceph.mon.a.smithi089.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: starting mon.a rank 0 at public addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] at bind addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???) e1 preinit fsid 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???).mds e1 new map 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???).mds e1 print_map 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: e1 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: legacy client fscid: -1 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout: No filesystems configured 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:05:45.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 bash[30057]: 7c27706dcbfb3d96e18d3a1220b981f232d12ddc40388c94ba5f816d8cae4954 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 systemd[1]: Started Ceph mon.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: monmap e1: 1 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: fsmap 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T13:05:45.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:45 smithi089 ceph-mon[30095]: mgrmap e1: no daemons active 2024-03-21T13:05:47.052 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.467+0000 7f93ac55e700 1 Processor -- start 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.467+0000 7f93ac55e700 1 -- start start 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.467+0000 7f93ac55e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a4104bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.467+0000 7f93ac55e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a4079640 con 0x7f93a4104790 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93aa2fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a4104bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93aa2fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a4104bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51106/0 (socket says 172.21.15.89:51106) 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93aa2fa700 1 -- 172.21.15.89:0/2723788219 learned_addr learned my addr 172.21.15.89:0/2723788219 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93aa2fa700 1 -- 172.21.15.89:0/2723788219 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a41050f0 con 0x7f93a4104790 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93aa2fa700 1 --2- 172.21.15.89:0/2723788219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a4104bb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93a00098a0 tx=0x7f93a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421b02e56e4b4f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93a92f8700 1 -- 172.21.15.89:0/2723788219 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a0004160 con 0x7f93a4104790 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93a92f8700 1 -- 172.21.15.89:0/2723788219 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f93a0004300 con 0x7f93a4104790 2024-03-21T13:05:47.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.468+0000 7f93a92f8700 1 -- 172.21.15.89:0/2723788219 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a0011940 con 0x7f93a4104790 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.469+0000 7f93ac55e700 1 -- 172.21.15.89:0/2723788219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 msgr2=0x7f93a4104bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.469+0000 7f93ac55e700 1 --2- 172.21.15.89:0/2723788219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a4104bb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93a00098a0 tx=0x7f93a0009bb0 comp rx=0 tx=0).stop 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.469+0000 7f93ac55e700 1 -- 172.21.15.89:0/2723788219 shutdown_connections 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.469+0000 7f93ac55e700 1 --2- 172.21.15.89:0/2723788219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a4104bb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.469+0000 7f93ac55e700 1 -- 172.21.15.89:0/2723788219 >> 172.21.15.89:0/2723788219 conn(0x7f93a4100370 msgr2=0x7f93a4102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.469+0000 7f93ac55e700 1 -- 172.21.15.89:0/2723788219 shutdown_connections 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.469+0000 7f93ac55e700 1 -- 172.21.15.89:0/2723788219 wait complete. 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93ac55e700 1 Processor -- start 2024-03-21T13:05:47.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93ac55e700 1 -- start start 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93ac55e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a419bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93ac55e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a4106060 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93aa2fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a419bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93aa2fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a419bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51120/0 (socket says 172.21.15.89:51120) 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93aa2fa700 1 -- 172.21.15.89:0/638490015 learned_addr learned my addr 172.21.15.89:0/638490015 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93aa2fa700 1 -- 172.21.15.89:0/638490015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a419c500 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93aa2fa700 1 --2- 172.21.15.89:0/638490015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a419bfc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009290 tx=0x7f93a00119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f939b7fe700 1 -- 172.21.15.89:0/638490015 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a0003930 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f939b7fe700 1 -- 172.21.15.89:0/638490015 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f93a0003ad0 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.470+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a419c790 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.471+0000 7f939b7fe700 1 -- 172.21.15.89:0/638490015 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a0019720 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.471+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a419f430 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.471+0000 7f939b7fe700 1 -- 172.21.15.89:0/638490015 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f93a002d440 con 0x7f93a4104790 2024-03-21T13:05:47.060 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.471+0000 7f939b7fe700 1 -- 172.21.15.89:0/638490015 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f93a0024850 con 0x7f93a4104790 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.471+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a40637c0 con 0x7f93a4104790 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.474+0000 7f939b7fe700 1 -- 172.21.15.89:0/638490015 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f93a0043dd0 con 0x7f93a4104790 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.539+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f93a402daa0 con 0x7f93a4104790 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.541+0000 7f939b7fe700 1 -- 172.21.15.89:0/638490015 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f93a0032020 con 0x7f93a4104790 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.542+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 msgr2=0x7f93a419bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.542+0000 7f93ac55e700 1 --2- 172.21.15.89:0/638490015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a419bfc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009290 tx=0x7f93a00119c0 comp rx=0 tx=0).stop 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.542+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 shutdown_connections 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.542+0000 7f93ac55e700 1 --2- 172.21.15.89:0/638490015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4104790 0x7f93a419bfc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.542+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 >> 172.21.15.89:0/638490015 conn(0x7f93a4100370 msgr2=0x7f93a4192c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.542+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 shutdown_connections 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:46.542+0000 7f93ac55e700 1 -- 172.21.15.89:0/638490015 wait complete. 2024-03-21T13:05:47.061 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-21T13:05:47.074 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:05:47.074 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mgr... 2024-03-21T13:05:47.075 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-21T13:05:47.075 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-21T13:05:47.320 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y 2024-03-21T13:05:47.320 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-21T13:05:47.469 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9c355834-e783-11ee-95cd-87774f69a715.target.wants/ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-9c355834-e783-11ee-95cd-87774f69a715@.service. 2024-03-21T13:05:47.818 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/638490015' entity='client.admin' 2024-03-21T13:05:47.818 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:47 smithi089 podman[30336]: 2024-03-21 13:05:47.606663106 +0000 UTC m=+0.017357144 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:48.042 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:05:48.043 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-03-21T13:05:48.049 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:05:48.050 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-03-21T13:05:48.050 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-03-21T13:05:48.050 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T13:05:48.058 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:05:48.058 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-03-21T13:05:48.065 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:05:48.065 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-03-21T13:05:48.065 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-03-21T13:05:48.065 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-21T13:05:48.065 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr to start... 2024-03-21T13:05:48.066 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr... 2024-03-21T13:05:48.188 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:47 smithi089 podman[30336]: 2024-03-21 13:05:47.920549185 +0000 UTC m=+0.331243216 container init ca14242b4eecdc9c5c1c1294ce98baacc4ad9c13a3e7b2a41ccfea5a97322341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-21T13:05:48.188 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:47 smithi089 podman[30336]: 2024-03-21 13:05:47.925868551 +0000 UTC m=+0.336562587 container start ca14242b4eecdc9c5c1c1294ce98baacc4ad9c13a3e7b2a41ccfea5a97322341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=) 2024-03-21T13:05:48.188 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:48 smithi089 bash[30336]: ca14242b4eecdc9c5c1c1294ce98baacc4ad9c13a3e7b2a41ccfea5a97322341 2024-03-21T13:05:48.188 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:48 smithi089 systemd[1]: Started Ceph mgr.y for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:05:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1871364786' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:05:48.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:48 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:48.484+0000 7f544a12a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:05:48.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:48 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:48.582+0000 7f544a12a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:05:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:05:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "9c355834-e783-11ee-95cd-87774f69a715", 2024-03-21T13:05:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:05:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:05:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:05:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:05:49.174 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:05:49.175 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:05:49.182 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:49.182 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:05:49.182 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T13:05:49.183 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:05:40.974589+0000", 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:05:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.534+0000 7fd9e0089700 1 Processor -- start 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.534+0000 7fd9e0089700 1 -- start start 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.534+0000 7fd9e0089700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8108d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.535+0000 7fd9e0089700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8079680 con 0x7fd9d8106940 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.535+0000 7fd9dde25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8108d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.535+0000 7fd9dde25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8108d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43868/0 (socket says 172.21.15.89:43868) 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.535+0000 7fd9dde25700 1 -- 172.21.15.89:0/1454645710 learned_addr learned my addr 172.21.15.89:0/1454645710 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.535+0000 7fd9dde25700 1 -- 172.21.15.89:0/1454645710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d8109270 con 0x7fd9d8106940 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.535+0000 7fd9dde25700 1 --2- 172.21.15.89:0/1454645710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8108d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40098a0 tx=0x7fd9d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b44b78f2c13fc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.535+0000 7fd9dce23700 1 -- 172.21.15.89:0/1454645710 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9d4004160 con 0x7fd9d8106940 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.536+0000 7fd9dce23700 1 -- 172.21.15.89:0/1454645710 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd9d4004300 con 0x7fd9d8106940 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.536+0000 7fd9dce23700 1 -- 172.21.15.89:0/1454645710 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9d4011940 con 0x7fd9d8106940 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.536+0000 7fd9e0089700 1 -- 172.21.15.89:0/1454645710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 msgr2=0x7fd9d8108d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.536+0000 7fd9e0089700 1 --2- 172.21.15.89:0/1454645710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8108d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40098a0 tx=0x7fd9d4009bb0 comp rx=0 tx=0).stop 2024-03-21T13:05:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.536+0000 7fd9e0089700 1 -- 172.21.15.89:0/1454645710 shutdown_connections 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.536+0000 7fd9e0089700 1 --2- 172.21.15.89:0/1454645710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8108d30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.536+0000 7fd9e0089700 1 -- 172.21.15.89:0/1454645710 >> 172.21.15.89:0/1454645710 conn(0x7fd9d8100310 msgr2=0x7fd9d8102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.537+0000 7fd9e0089700 1 -- 172.21.15.89:0/1454645710 shutdown_connections 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.537+0000 7fd9e0089700 1 -- 172.21.15.89:0/1454645710 wait complete. 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.537+0000 7fd9e0089700 1 Processor -- start 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.537+0000 7fd9e0089700 1 -- start start 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.537+0000 7fd9e0089700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9e0089700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d810a170 con 0x7fd9d8106940 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9dde25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9dde25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43872/0 (socket says 172.21.15.89:43872) 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9dde25700 1 -- 172.21.15.89:0/1871364786 learned_addr learned my addr 172.21.15.89:0/1871364786 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9dde25700 1 -- 172.21.15.89:0/1871364786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d819a220 con 0x7fd9d8106940 2024-03-21T13:05:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9dde25700 1 --2- 172.21.15.89:0/1871364786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8199ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4003e40 tx=0x7fd9d40119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9ceffd700 1 -- 172.21.15.89:0/1871364786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9d4003b20 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9ceffd700 1 -- 172.21.15.89:0/1871364786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd9d401a890 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9ceffd700 1 -- 172.21.15.89:0/1871364786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9d4024890 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d819a4b0 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.538+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d819d190 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.539+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d80637c0 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.539+0000 7fd9ceffd700 1 -- 172.21.15.89:0/1871364786 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd9d4022020 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.539+0000 7fd9ceffd700 1 -- 172.21.15.89:0/1871364786 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd9d402da60 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.541+0000 7fd9ceffd700 1 -- 172.21.15.89:0/1871364786 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd9d401aa40 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.615+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd9d819d5f0 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.615+0000 7fd9ceffd700 1 -- 172.21.15.89:0/1871364786 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd9d401ad90 con 0x7fd9d8106940 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.616+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 msgr2=0x7fd9d8199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.616+0000 7fd9e0089700 1 --2- 172.21.15.89:0/1871364786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8199ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4003e40 tx=0x7fd9d40119c0 comp rx=0 tx=0).stop 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.616+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 shutdown_connections 2024-03-21T13:05:49.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.616+0000 7fd9e0089700 1 --2- 172.21.15.89:0/1871364786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8106940 0x7fd9d8199ce0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:49.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.616+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 >> 172.21.15.89:0/1871364786 conn(0x7fd9d8100310 msgr2=0x7fd9d8100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:49.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.616+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 shutdown_connections 2024-03-21T13:05:49.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:48.616+0000 7fd9e0089700 1 -- 172.21.15.89:0/1871364786 wait complete. 2024-03-21T13:05:49.188 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (1/15)... 2024-03-21T13:05:49.562 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:49.235+0000 7f544a12a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:05:49.562 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:49.327+0000 7f544a12a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:05:49.562 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:49.467+0000 7f544a12a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:05:49.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:49.561+0000 7f544a12a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:05:49.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:49.643+0000 7f544a12a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:05:50.199 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:49.910+0000 7f544a12a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:05:50.199 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:50 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:50.019+0000 7f544a12a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:05:50.199 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:50 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:50.105+0000 7f544a12a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:05:50.199 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:50 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:50.198+0000 7f544a12a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:05:50.574 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:50 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:50.371+0000 7f544a12a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:05:51.562 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:51.344+0000 7f544a12a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:05:51.562 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:51.480+0000 7f544a12a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:05:51.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2822530121' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:05:52.106 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:51.824+0000 7f544a12a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:05:52.106 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:51.913+0000 7f544a12a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:05:52.106 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:52.105+0000 7f544a12a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:05:52.499 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:52.314+0000 7f544a12a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:05:52.503 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:52.503 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:05:52.503 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "9c355834-e783-11ee-95cd-87774f69a715", 2024-03-21T13:05:52.503 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:05:52.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:52.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:05:52.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:05:52.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:52.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:05:52.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:52.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:05:52.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:05:52.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T13:05:52.514 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:05:40.974589+0000", 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:05:52.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7febad334700 1 Processor -- start 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7febad334700 1 -- start start 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7febad334700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7febad334700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80795e0 con 0x7feba8106a20 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7feba6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7feba6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43882/0 (socket says 172.21.15.89:43882) 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7feba6ffd700 1 -- 172.21.15.89:0/121888956 learned_addr learned my addr 172.21.15.89:0/121888956 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.666+0000 7feba6ffd700 1 -- 172.21.15.89:0/121888956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8109350 con 0x7feba8106a20 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.667+0000 7feba6ffd700 1 --2- 172.21.15.89:0/121888956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8108e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb980098a0 tx=0x7feb98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7523dbb435cca014 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.667+0000 7feba5ffb700 1 -- 172.21.15.89:0/121888956 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb98004160 con 0x7feba8106a20 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.667+0000 7feba5ffb700 1 -- 172.21.15.89:0/121888956 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7feb98004300 con 0x7feba8106a20 2024-03-21T13:05:52.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.667+0000 7feba5ffb700 1 -- 172.21.15.89:0/121888956 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb98011940 con 0x7feba8106a20 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.667+0000 7febad334700 1 -- 172.21.15.89:0/121888956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 msgr2=0x7feba8108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.667+0000 7febad334700 1 --2- 172.21.15.89:0/121888956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8108e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb980098a0 tx=0x7feb98009bb0 comp rx=0 tx=0).stop 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 -- 172.21.15.89:0/121888956 shutdown_connections 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 --2- 172.21.15.89:0/121888956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8108e10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 -- 172.21.15.89:0/121888956 >> 172.21.15.89:0/121888956 conn(0x7feba8100370 msgr2=0x7feba81027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 -- 172.21.15.89:0/121888956 shutdown_connections 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 -- 172.21.15.89:0/121888956 wait complete. 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 Processor -- start 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 -- start start 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.668+0000 7febad334700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7febad334700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba810a180 con 0x7feba8106a20 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feba6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feba6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43894/0 (socket says 172.21.15.89:43894) 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feba6ffd700 1 -- 172.21.15.89:0/2822530121 learned_addr learned my addr 172.21.15.89:0/2822530121 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feba6ffd700 1 -- 172.21.15.89:0/2822530121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba819a2e0 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feba6ffd700 1 --2- 172.21.15.89:0/2822530121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8199da0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb98009290 tx=0x7feb980119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feb8ffff700 1 -- 172.21.15.89:0/2822530121 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb98003930 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba819a570 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feb8ffff700 1 -- 172.21.15.89:0/2822530121 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7feb98003ad0 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7feb8ffff700 1 -- 172.21.15.89:0/2822530121 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb98019720 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.669+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba819d220 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.670+0000 7feb8ffff700 1 -- 172.21.15.89:0/2822530121 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7feb9802d440 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.670+0000 7feb8ffff700 1 -- 172.21.15.89:0/2822530121 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feb98024840 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.670+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba80637c0 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.673+0000 7feb8ffff700 1 -- 172.21.15.89:0/2822530121 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7feb98043dd0 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.746+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feba819d600 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.747+0000 7feb8ffff700 1 -- 172.21.15.89:0/2822530121 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7feb98032030 con 0x7feba8106a20 2024-03-21T13:05:52.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.748+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 msgr2=0x7feba8199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:52.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.748+0000 7febad334700 1 --2- 172.21.15.89:0/2822530121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8199da0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb98009290 tx=0x7feb980119c0 comp rx=0 tx=0).stop 2024-03-21T13:05:52.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.748+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 shutdown_connections 2024-03-21T13:05:52.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.748+0000 7febad334700 1 --2- 172.21.15.89:0/2822530121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba8106a20 0x7feba8199da0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:52.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.748+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 >> 172.21.15.89:0/2822530121 conn(0x7feba8100370 msgr2=0x7feba8100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:52.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.748+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 shutdown_connections 2024-03-21T13:05:52.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:51.748+0000 7febad334700 1 -- 172.21.15.89:0/2822530121 wait complete. 2024-03-21T13:05:52.519 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (2/15)... 2024-03-21T13:05:52.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:52.598+0000 7f544a12a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:05:53.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:52.854+0000 7f544a12a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:05:53.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:53 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:53.086+0000 7f544a12a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:05:54.324 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:53 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:53.887+0000 7f544a12a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:05:54.748 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:54.447+0000 7f544a12a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:05:54.748 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:54.547+0000 7f544a12a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:05:54.999 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:05:54.768+0000 7f544a12a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: Activating manager daemon y 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: mgrmap e2: y(active, starting, since 0.00304974s) 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: Manager daemon y is now available 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:05:55.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' 2024-03-21T13:05:55.001 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' 2024-03-21T13:05:55.001 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:54 smithi089 ceph-mon[30095]: from='mgr.14100 172.21.15.89:0/259228867' entity='mgr.y' 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "9c355834-e783-11ee-95cd-87774f69a715", 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:05:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:05:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:05:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:05:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:05:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:05:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:05:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:05:55.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:05:40.974589+0000", 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:05:55.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86b1d21700 1 Processor -- start 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86b1d21700 1 -- start start 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86b1d21700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac1078e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86b1d21700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac107eb0 con 0x7f86ac1074c0 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86ab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac1078e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86ab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac1078e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43974/0 (socket says 172.21.15.89:43974) 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86ab7fe700 1 -- 172.21.15.89:0/1767351641 learned_addr learned my addr 172.21.15.89:0/1767351641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:54.999+0000 7f86ab7fe700 1 -- 172.21.15.89:0/1767351641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac108030 con 0x7f86ac1074c0 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.000+0000 7f86ab7fe700 1 --2- 172.21.15.89:0/1767351641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac1078e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f869c0098a0 tx=0x7f869c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8882b33857ba874 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.000+0000 7f86aa7fc700 1 -- 172.21.15.89:0/1767351641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c004160 con 0x7f86ac1074c0 2024-03-21T13:05:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.000+0000 7f86aa7fc700 1 -- 172.21.15.89:0/1767351641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f869c004300 con 0x7f86ac1074c0 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.000+0000 7f86aa7fc700 1 -- 172.21.15.89:0/1767351641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c011940 con 0x7f86ac1074c0 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.000+0000 7f86b1d21700 1 -- 172.21.15.89:0/1767351641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 msgr2=0x7f86ac1078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.000+0000 7f86b1d21700 1 --2- 172.21.15.89:0/1767351641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac1078e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f869c0098a0 tx=0x7f869c009bb0 comp rx=0 tx=0).stop 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.001+0000 7f86b1d21700 1 -- 172.21.15.89:0/1767351641 shutdown_connections 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.001+0000 7f86b1d21700 1 --2- 172.21.15.89:0/1767351641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac1078e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.001+0000 7f86b1d21700 1 -- 172.21.15.89:0/1767351641 >> 172.21.15.89:0/1767351641 conn(0x7f86ac0ffae0 msgr2=0x7f86ac101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.001+0000 7f86b1d21700 1 -- 172.21.15.89:0/1767351641 shutdown_connections 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.001+0000 7f86b1d21700 1 -- 172.21.15.89:0/1767351641 wait complete. 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.001+0000 7f86b1d21700 1 Processor -- start 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.001+0000 7f86b1d21700 1 -- start start 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86b1d21700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86b1d21700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac104c10 con 0x7f86ac1074c0 2024-03-21T13:05:55.590 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86ab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86ab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43984/0 (socket says 172.21.15.89:43984) 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86ab7fe700 1 -- 172.21.15.89:0/1231952267 learned_addr learned my addr 172.21.15.89:0/1231952267 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86ab7fe700 1 -- 172.21.15.89:0/1231952267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac19a2e0 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86ab7fe700 1 --2- 172.21.15.89:0/1231952267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac199da0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f869c006ba0 tx=0x7f869c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86a8ff9700 1 -- 172.21.15.89:0/1231952267 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c003930 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86a8ff9700 1 -- 172.21.15.89:0/1231952267 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f869c003ad0 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac19a5d0 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86a8ff9700 1 -- 172.21.15.89:0/1231952267 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c019750 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.002+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac19d220 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.003+0000 7f86a8ff9700 1 -- 172.21.15.89:0/1231952267 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 2) v1 ==== 46050+0+0 (secure 0 0 0) 0x7f869c0199c0 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.003+0000 7f86a8ff9700 1 -- 172.21.15.89:0/1231952267 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f869c04e120 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.003+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac0637c0 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.006+0000 7f86a8ff9700 1 -- 172.21.15.89:0/1231952267 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f869c020240 con 0x7f86ac1074c0 2024-03-21T13:05:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.075+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f86ac19d5d0 con 0x7f86ac1074c0 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.076+0000 7f86a8ff9700 1 -- 172.21.15.89:0/1231952267 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f869c01abe0 con 0x7f86ac1074c0 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.077+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 msgr2=0x7f86ac199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.077+0000 7f86b1d21700 1 --2- 172.21.15.89:0/1231952267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac199da0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f869c006ba0 tx=0x7f869c0119c0 comp rx=0 tx=0).stop 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.077+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 shutdown_connections 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.077+0000 7f86b1d21700 1 --2- 172.21.15.89:0/1231952267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ac1074c0 0x7f86ac199da0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.077+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 >> 172.21.15.89:0/1231952267 conn(0x7f86ac0ffae0 msgr2=0x7f86ac100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.077+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 shutdown_connections 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:05:55.077+0000 7f86b1d21700 1 -- 172.21.15.89:0/1231952267 wait complete. 2024-03-21T13:05:55.592 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (3/15)... 2024-03-21T13:05:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1231952267' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:05:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:55 smithi089 ceph-mon[30095]: mgrmap e3: y(active, since 1.00704s) 2024-03-21T13:05:58.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:05:57 smithi089 ceph-mon[30095]: mgrmap e4: y(active, since 2s) 2024-03-21T13:06:01.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1789397937' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:06:07.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "9c355834-e783-11ee-95cd-87774f69a715", 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:06:07.719 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:06:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:06:07.721 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:06:07.722 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:05:40.974589+0000", 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce0aa9c700 1 Processor -- start 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce0aa9c700 1 -- start start 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce0aa9c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce04079b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce0aa9c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0407a0e0 con 0x7fce0407b6b0 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce08838700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce04079b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:07.723 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce08838700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce04079b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58204/0 (socket says 172.21.15.89:58204) 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce08838700 1 -- 172.21.15.89:0/2342681139 learned_addr learned my addr 172.21.15.89:0/2342681139 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.803+0000 7fce08838700 1 -- 172.21.15.89:0/2342681139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0407a260 con 0x7fce0407b6b0 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce08838700 1 --2- 172.21.15.89:0/2342681139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce04079b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80098a0 tx=0x7fcdf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1991a5606bba8b46 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce037fe700 1 -- 172.21.15.89:0/2342681139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdf8004160 con 0x7fce0407b6b0 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce037fe700 1 -- 172.21.15.89:0/2342681139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fcdf8004300 con 0x7fce0407b6b0 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce037fe700 1 -- 172.21.15.89:0/2342681139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdf8011940 con 0x7fce0407b6b0 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce0aa9c700 1 -- 172.21.15.89:0/2342681139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 msgr2=0x7fce04079b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce0aa9c700 1 --2- 172.21.15.89:0/2342681139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce04079b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80098a0 tx=0x7fcdf8009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce0aa9c700 1 -- 172.21.15.89:0/2342681139 shutdown_connections 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce0aa9c700 1 --2- 172.21.15.89:0/2342681139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce04079b10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce0aa9c700 1 -- 172.21.15.89:0/2342681139 >> 172.21.15.89:0/2342681139 conn(0x7fce041014b0 msgr2=0x7fce041038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce0aa9c700 1 -- 172.21.15.89:0/2342681139 shutdown_connections 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.804+0000 7fce0aa9c700 1 -- 172.21.15.89:0/2342681139 wait complete. 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce0aa9c700 1 Processor -- start 2024-03-21T13:06:07.724 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce0aa9c700 1 -- start start 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce0aa9c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce0419bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce0aa9c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0410c3e0 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce08838700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce0419bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce08838700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce0419bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58216/0 (socket says 172.21.15.89:58216) 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce08838700 1 -- 172.21.15.89:0/1789397937 learned_addr learned my addr 172.21.15.89:0/1789397937 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.805+0000 7fce08838700 1 -- 172.21.15.89:0/1789397937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0419c530 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.806+0000 7fce08838700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce0419bff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8011910 tx=0x7fcdf80119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.806+0000 7fce01ffb700 1 -- 172.21.15.89:0/1789397937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdf8003b20 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.806+0000 7fce01ffb700 1 -- 172.21.15.89:0/1789397937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fcdf801a890 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.806+0000 7fce01ffb700 1 -- 172.21.15.89:0/1789397937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdf80197f0 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.806+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0419c7c0 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.806+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0419f4b0 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.807+0000 7fce01ffb700 1 -- 172.21.15.89:0/1789397937 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 46283+0+0 (secure 0 0 0) 0x7fcdf8022020 con 0x7fce0407b6b0 2024-03-21T13:06:07.725 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.807+0000 7fce01ffb700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fcdf4039180 0x7fcdf403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.807+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce040637c0 con 0x7fce0407b6b0 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.807+0000 7fce01ffb700 1 -- 172.21.15.89:0/1789397937 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcdf804d2d0 con 0x7fce0407b6b0 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.807+0000 7fce03fff700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fcdf4039180 0x7fcdf403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.808+0000 7fce03fff700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fcdf4039180 0x7fcdf403b640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0006fd0 tx=0x7fcdf0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:00.812+0000 7fce01ffb700 1 -- 172.21.15.89:0/1789397937 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcdf802b350 con 0x7fce0407b6b0 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.089+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fce0419f910 con 0x7fce0407b6b0 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.089+0000 7fce01ffb700 1 -- 172.21.15.89:0/1789397937 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fcdf801ad90 con 0x7fce0407b6b0 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.091+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fcdf4039180 msgr2=0x7fcdf403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.091+0000 7fce0aa9c700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fcdf4039180 0x7fcdf403b640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0006fd0 tx=0x7fcdf0006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.091+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 msgr2=0x7fce0419bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.091+0000 7fce0aa9c700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce0419bff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8011910 tx=0x7fcdf80119f0 comp rx=0 tx=0).stop 2024-03-21T13:06:07.726 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.092+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 shutdown_connections 2024-03-21T13:06:07.727 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.092+0000 7fce0aa9c700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fcdf4039180 0x7fcdf403b640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:07.727 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.092+0000 7fce0aa9c700 1 --2- 172.21.15.89:0/1789397937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce0407b6b0 0x7fce0419bff0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:07.727 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.092+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 >> 172.21.15.89:0/1789397937 conn(0x7fce041014b0 msgr2=0x7fce04102010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:07.727 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.092+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 shutdown_connections 2024-03-21T13:06:07.727 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:01.092+0000 7fce0aa9c700 1 -- 172.21.15.89:0/1789397937 wait complete. 2024-03-21T13:06:07.727 INFO:teuthology.orchestra.run.smithi089.stdout:mgr is available 2024-03-21T13:06:08.911 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2188031991' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T13:06:09.196 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:06:09.196 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T13:06:09.196 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T13:06:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.368+0000 7f1ff4129700 1 Processor -- start 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.368+0000 7f1ff4129700 1 -- start start 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.368+0000 7f1ff4129700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec104930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.368+0000 7f1ff4129700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec104f00 con 0x7f1fec104510 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.368+0000 7f1ff1ec5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec104930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.369+0000 7f1ff1ec5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec104930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58032/0 (socket says 172.21.15.89:58032) 2024-03-21T13:06:09.198 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.369+0000 7f1ff1ec5700 1 -- 172.21.15.89:0/15643428 learned_addr learned my addr 172.21.15.89:0/15643428 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.369+0000 7f1ff1ec5700 1 -- 172.21.15.89:0/15643428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec105710 con 0x7f1fec104510 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.369+0000 7f1ff1ec5700 1 --2- 172.21.15.89:0/15643428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec104930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098a0 tx=0x7f1fdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0d88e8a8f93d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.369+0000 7f1ff0ec3700 1 -- 172.21.15.89:0/15643428 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fdc004160 con 0x7f1fec104510 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.369+0000 7f1ff0ec3700 1 -- 172.21.15.89:0/15643428 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f1fdc004300 con 0x7f1fec104510 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.369+0000 7f1ff0ec3700 1 -- 172.21.15.89:0/15643428 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fdc011940 con 0x7f1fec104510 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.370+0000 7f1ff4129700 1 -- 172.21.15.89:0/15643428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 msgr2=0x7f1fec104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.370+0000 7f1ff4129700 1 --2- 172.21.15.89:0/15643428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec104930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098a0 tx=0x7f1fdc009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.370+0000 7f1ff4129700 1 -- 172.21.15.89:0/15643428 shutdown_connections 2024-03-21T13:06:09.199 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.370+0000 7f1ff4129700 1 --2- 172.21.15.89:0/15643428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec104930 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.370+0000 7f1ff4129700 1 -- 172.21.15.89:0/15643428 >> 172.21.15.89:0/15643428 conn(0x7f1fec0ffa90 msgr2=0x7f1fec101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.370+0000 7f1ff4129700 1 -- 172.21.15.89:0/15643428 shutdown_connections 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.370+0000 7f1ff4129700 1 -- 172.21.15.89:0/15643428 wait complete. 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff4129700 1 Processor -- start 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff4129700 1 -- start start 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff4129700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec18f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff4129700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec108940 con 0x7f1fec104510 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff1ec5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec18f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:09.200 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff1ec5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec18f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58034/0 (socket says 172.21.15.89:58034) 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff1ec5700 1 -- 172.21.15.89:0/2188031991 learned_addr learned my addr 172.21.15.89:0/2188031991 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff1ec5700 1 -- 172.21.15.89:0/2188031991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec18faf0 con 0x7f1fec104510 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.371+0000 7f1ff1ec5700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec18f5b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc003e40 tx=0x7f1fdc0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.372+0000 7f1fe2ffd700 1 -- 172.21.15.89:0/2188031991 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fdc01a430 con 0x7f1fec104510 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.372+0000 7f1fe2ffd700 1 -- 172.21.15.89:0/2188031991 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f1fdc01aa30 con 0x7f1fec104510 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.372+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec18dce0 con 0x7f1fec104510 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.372+0000 7f1fe2ffd700 1 -- 172.21.15.89:0/2188031991 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fdc024960 con 0x7f1fec104510 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.372+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec18e220 con 0x7f1fec104510 2024-03-21T13:06:09.201 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.372+0000 7f1fe2ffd700 1 -- 172.21.15.89:0/2188031991 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 46283+0+0 (secure 0 0 0) 0x7f1fdc022020 con 0x7f1fec104510 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.373+0000 7f1fe2ffd700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f1fd80390e0 0x7f1fd803b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.373+0000 7f1fe2ffd700 1 -- 172.21.15.89:0/2188031991 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1fdc04d5e0 con 0x7f1fec104510 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.373+0000 7f1ff16c4700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f1fd80390e0 0x7f1fd803b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.373+0000 7f1ff16c4700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f1fd80390e0 0x7f1fd803b5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8006fd0 tx=0x7f1fe8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.373+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec108ac0 con 0x7f1fec104510 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.378+0000 7f1fe2ffd700 1 -- 172.21.15.89:0/2188031991 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1fdc00b3e0 con 0x7f1fec104510 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.598+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1fec0637c0 con 0x7f1fec104510 2024-03-21T13:06:09.202 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.599+0000 7f1fe2ffd700 1 -- 172.21.15.89:0/2188031991 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f1fdc004300 con 0x7f1fec104510 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f1fd80390e0 msgr2=0x7f1fd803b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f1fd80390e0 0x7f1fd803b5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8006fd0 tx=0x7f1fe8006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 msgr2=0x7f1fec18f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec18f5b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc003e40 tx=0x7f1fdc0119c0 comp rx=0 tx=0).stop 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 shutdown_connections 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f1fd80390e0 0x7f1fd803b5a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 --2- 172.21.15.89:0/2188031991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fec104510 0x7f1fec18f5b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 >> 172.21.15.89:0/2188031991 conn(0x7f1fec0ffa90 msgr2=0x7f1fec100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 shutdown_connections 2024-03-21T13:06:09.203 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:08.601+0000 7f1ff4129700 1 -- 172.21.15.89:0/2188031991 wait complete. 2024-03-21T13:06:09.204 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling cephadm module... 2024-03-21T13:06:10.200 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/755983686' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-21T13:06:10.200 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:09 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ignoring --setuser ceph since I am not root 2024-03-21T13:06:10.200 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:09 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ignoring --setgroup ceph since I am not root 2024-03-21T13:06:10.484 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.590+0000 7f36950d0700 1 Processor -- start 2024-03-21T13:06:10.484 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.590+0000 7f36950d0700 1 -- start start 2024-03-21T13:06:10.484 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.590+0000 7f36950d0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f3690079af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.590+0000 7f36950d0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369007a0c0 con 0x7f369007b690 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.590+0000 7f368ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f3690079af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.590+0000 7f368ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f3690079af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58048/0 (socket says 172.21.15.89:58048) 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.590+0000 7f368ed9d700 1 -- 172.21.15.89:0/271282073 learned_addr learned my addr 172.21.15.89:0/271282073 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.591+0000 7f368ed9d700 1 -- 172.21.15.89:0/271282073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369007a240 con 0x7f369007b690 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.591+0000 7f368ed9d700 1 --2- 172.21.15.89:0/271282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f3690079af0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f36800098a0 tx=0x7f3680009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335d688ea581dcad server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.591+0000 7f368e59c700 1 -- 172.21.15.89:0/271282073 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3680004160 con 0x7f369007b690 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.591+0000 7f368e59c700 1 -- 172.21.15.89:0/271282073 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f3680004300 con 0x7f369007b690 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.591+0000 7f368e59c700 1 -- 172.21.15.89:0/271282073 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3680011940 con 0x7f369007b690 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.591+0000 7f36950d0700 1 -- 172.21.15.89:0/271282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 msgr2=0x7f3690079af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.591+0000 7f36950d0700 1 --2- 172.21.15.89:0/271282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f3690079af0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f36800098a0 tx=0x7f3680009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 -- 172.21.15.89:0/271282073 shutdown_connections 2024-03-21T13:06:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 --2- 172.21.15.89:0/271282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f3690079af0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 -- 172.21.15.89:0/271282073 >> 172.21.15.89:0/271282073 conn(0x7f3690101460 msgr2=0x7f36901038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 -- 172.21.15.89:0/271282073 shutdown_connections 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 -- 172.21.15.89:0/271282073 wait complete. 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 Processor -- start 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 -- start start 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f369019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.592+0000 7f36950d0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369010c3d0 con 0x7f369007b690 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f369019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f369019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58050/0 (socket says 172.21.15.89:58050) 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368ed9d700 1 -- 172.21.15.89:0/755983686 learned_addr learned my addr 172.21.15.89:0/755983686 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368ed9d700 1 -- 172.21.15.89:0/755983686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369019c480 con 0x7f369007b690 2024-03-21T13:06:10.486 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368ed9d700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f369019bf40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3680011910 tx=0x7f36800119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3680003ac0 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369019c710 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f368001a890 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36800197a0 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.593+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369019f400 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.594+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 46283+0+0 (secure 0 0 0) 0x7f3680019aa0 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.594+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36900637c0 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.594+0000 7f368cd99700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f367c039130 0x7f367c03b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.594+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f368004dc70 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.594+0000 7f3687fff700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f367c039130 0x7f367c03b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.595+0000 7f3687fff700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f367c039130 0x7f367c03b5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3678006fd0 tx=0x7f3678006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.599+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3680020240 con 0x7f369007b690 2024-03-21T13:06:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.846+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f369019f860 con 0x7f369007b690 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.898+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f368001ad90 con 0x7f369007b690 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.899+0000 7f368cd99700 1 -- 172.21.15.89:0/755983686 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 5) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f368002d960 con 0x7f369007b690 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.900+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f367c039130 msgr2=0x7f367c03b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.900+0000 7f36950d0700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f367c039130 0x7f367c03b5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3678006fd0 tx=0x7f3678006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.900+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 msgr2=0x7f369019bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.900+0000 7f36950d0700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f369019bf40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3680011910 tx=0x7f36800119f0 comp rx=0 tx=0).stop 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.900+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 shutdown_connections 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.900+0000 7f36950d0700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f367c039130 0x7f367c03b5f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.901+0000 7f36950d0700 1 --2- 172.21.15.89:0/755983686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f369007b690 0x7f369019bf40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.901+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 >> 172.21.15.89:0/755983686 conn(0x7f3690101460 msgr2=0x7f3690101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.901+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 shutdown_connections 2024-03-21T13:06:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:09.901+0000 7f36950d0700 1 -- 172.21.15.89:0/755983686 wait complete. 2024-03-21T13:06:10.710 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:10 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:10.456+0000 7f371a0bd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:06:10.710 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:10 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:10.555+0000 7f371a0bd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:06:11.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/755983686' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-21T13:06:11.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:10 smithi089 ceph-mon[30095]: mgrmap e5: y(active, since 15s) 2024-03-21T13:06:11.450 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:11 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:11.211+0000 7f371a0bd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:06:11.450 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:11 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:11.306+0000 7f371a0bd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:06:11.450 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:11 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:11.449+0000 7f371a0bd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:06:11.741 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:11 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:11.544+0000 7f371a0bd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:06:11.741 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:11 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:11.626+0000 7f371a0bd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:06:11.749 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:06:11.749 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-21T13:06:11.749 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.921+0000 7fb32177d700 1 Processor -- start 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.921+0000 7fb32177d700 1 -- start start 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.921+0000 7fb32177d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c108dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.921+0000 7fb32177d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c078d50 con 0x7fb31c1069e0 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb31affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c108dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb31affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c108dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58078/0 (socket says 172.21.15.89:58078) 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb31affd700 1 -- 172.21.15.89:0/4172018179 learned_addr learned my addr 172.21.15.89:0/4172018179 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb31affd700 1 -- 172.21.15.89:0/4172018179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c109310 con 0x7fb31c1069e0 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb31affd700 1 --2- 172.21.15.89:0/4172018179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c108dd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb3100098a0 tx=0x7fb310009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422d2047b89135a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:11.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb319ffb700 1 -- 172.21.15.89:0/4172018179 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb310004160 con 0x7fb31c1069e0 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb319ffb700 1 -- 172.21.15.89:0/4172018179 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fb310004300 con 0x7fb31c1069e0 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.922+0000 7fb319ffb700 1 -- 172.21.15.89:0/4172018179 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb310011940 con 0x7fb31c1069e0 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 -- 172.21.15.89:0/4172018179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 msgr2=0x7fb31c108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 --2- 172.21.15.89:0/4172018179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c108dd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb3100098a0 tx=0x7fb310009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 -- 172.21.15.89:0/4172018179 shutdown_connections 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 --2- 172.21.15.89:0/4172018179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c108dd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 -- 172.21.15.89:0/4172018179 >> 172.21.15.89:0/4172018179 conn(0x7fb31c100310 msgr2=0x7fb31c102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 -- 172.21.15.89:0/4172018179 shutdown_connections 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 -- 172.21.15.89:0/4172018179 wait complete. 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.923+0000 7fb32177d700 1 Processor -- start 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb32177d700 1 -- start start 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb32177d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb32177d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c10a140 con 0x7fb31c1069e0 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb31affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:11.751 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb31affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58094/0 (socket says 172.21.15.89:58094) 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb31affd700 1 -- 172.21.15.89:0/1134107500 learned_addr learned my addr 172.21.15.89:0/1134107500 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb31affd700 1 -- 172.21.15.89:0/1134107500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c19a260 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb31affd700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c199d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb310003e40 tx=0x7fb3100119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.924+0000 7fb303fff700 1 -- 172.21.15.89:0/1134107500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb310003b20 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.925+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c19a4f0 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.925+0000 7fb303fff700 1 -- 172.21.15.89:0/1134107500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fb31001a890 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.925+0000 7fb303fff700 1 -- 172.21.15.89:0/1134107500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb310024890 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.925+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c19d1d0 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.926+0000 7fb303fff700 1 -- 172.21.15.89:0/1134107500 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 46294+0+0 (secure 0 0 0) 0x7fb310022020 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.926+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c0637c0 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.926+0000 7fb303fff700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 0x7fb30403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.926+0000 7fb303fff700 1 -- 172.21.15.89:0/1134107500 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb31004d330 con 0x7fb31c1069e0 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.926+0000 7fb31a7fc700 1 -- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 msgr2=0x7fb30403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3099145381 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.926+0000 7fb31a7fc700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 0x7fb30403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:06:11.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:10.931+0000 7fb303fff700 1 -- 172.21.15.89:0/1134107500 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb310019e10 con 0x7fb31c1069e0 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.127+0000 7fb31a7fc700 1 -- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 msgr2=0x7fb30403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3099145381 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.127+0000 7fb31a7fc700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 0x7fb30403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.185+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb31c19d630 con 0x7fb31c1069e0 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.186+0000 7fb303fff700 1 -- 172.21.15.89:0/1134107500 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fb31002ce00 con 0x7fb31c1069e0 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 msgr2=0x7fb30403b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 0x7fb30403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 msgr2=0x7fb31c199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c199d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb310003e40 tx=0x7fb3100119c0 comp rx=0 tx=0).stop 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 shutdown_connections 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7fb304039180 0x7fb30403b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 --2- 172.21.15.89:0/1134107500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31c1069e0 0x7fb31c199d20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 >> 172.21.15.89:0/1134107500 conn(0x7fb31c100310 msgr2=0x7fb31c100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 shutdown_connections 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:11.188+0000 7fb32177d700 1 -- 172.21.15.89:0/1134107500 wait complete. 2024-03-21T13:06:11.753 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2024-03-21T13:06:11.754 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 5... 2024-03-21T13:06:11.996 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1134107500' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T13:06:11.996 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:11 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:11.896+0000 7f371a0bd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:06:12.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:12.013+0000 7f371a0bd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:06:12.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:12.099+0000 7f371a0bd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:06:12.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:12.199+0000 7f371a0bd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:06:12.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:12.377+0000 7f371a0bd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:06:13.629 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:13 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:13.325+0000 7f371a0bd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:06:13.629 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:13 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:13.454+0000 7f371a0bd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:06:14.068 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:13 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:13.785+0000 7f371a0bd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:06:14.068 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:13 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:13.874+0000 7f371a0bd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:06:14.068 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:14 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:14.067+0000 7f371a0bd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:06:14.324 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:14 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:14.250+0000 7f371a0bd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:06:14.785 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:14 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:14.530+0000 7f371a0bd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:06:14.785 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:14 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:14.784+0000 7f371a0bd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:06:15.074 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:15 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:15.020+0000 7f371a0bd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:06:16.074 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:15 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:15.813+0000 7f371a0bd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:06:16.684 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:16.373+0000 7f371a0bd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:06:16.685 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:16.471+0000 7f371a0bd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:06:16.685 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:16.684+0000 7f371a0bd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: Active manager daemon y restarted 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: Activating manager daemon y 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: osdmap e2: 0 total, 0 up, 0 in 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: mgrmap e6: y(active, starting, since 0.00337016s) 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:06:17.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:06:17.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:06:17.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: Manager daemon y is now available 2024-03-21T13:06:17.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:17.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:17.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:17.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:16 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:17.978 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:17 smithi089 ceph-mon[30095]: Found migration_current of "None". Setting to last migration. 2024-03-21T13:06:17.978 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:17 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:06:17.978 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:17 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:06:17.978 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:17 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:17.978 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:17 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:17.978 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:17 smithi089 ceph-mon[30095]: mgrmap e7: y(active, since 1.00564s) 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.221+0000 7f36e2dfa700 1 Processor -- start 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.221+0000 7f36e2dfa700 1 -- start start 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.221+0000 7f36e2dfa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc107890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.221+0000 7f36e2dfa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc107e60 con 0x7f36dc107470 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.221+0000 7f36e0b96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc107890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.221+0000 7f36e0b96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc107890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58098/0 (socket says 172.21.15.89:58098) 2024-03-21T13:06:18.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.221+0000 7f36e0b96700 1 -- 172.21.15.89:0/2492967137 learned_addr learned my addr 172.21.15.89:0/2492967137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.222+0000 7f36e0b96700 1 -- 172.21.15.89:0/2492967137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc107fe0 con 0x7f36dc107470 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.222+0000 7f36e0b96700 1 --2- 172.21.15.89:0/2492967137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc107890 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f36d00098a0 tx=0x7f36d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc55a3258340cbd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.222+0000 7f36db7fe700 1 -- 172.21.15.89:0/2492967137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36d0004160 con 0x7f36dc107470 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.222+0000 7f36db7fe700 1 -- 172.21.15.89:0/2492967137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f36d0004300 con 0x7f36dc107470 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.222+0000 7f36db7fe700 1 -- 172.21.15.89:0/2492967137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36d0011940 con 0x7f36dc107470 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 -- 172.21.15.89:0/2492967137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 msgr2=0x7f36dc107890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 --2- 172.21.15.89:0/2492967137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc107890 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f36d00098a0 tx=0x7f36d0009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 -- 172.21.15.89:0/2492967137 shutdown_connections 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 --2- 172.21.15.89:0/2492967137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc107890 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 -- 172.21.15.89:0/2492967137 >> 172.21.15.89:0/2492967137 conn(0x7f36dc0ffa90 msgr2=0x7f36dc101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 -- 172.21.15.89:0/2492967137 shutdown_connections 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 -- 172.21.15.89:0/2492967137 wait complete. 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.223+0000 7f36e2dfa700 1 Processor -- start 2024-03-21T13:06:18.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e2dfa700 1 -- start start 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e2dfa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e2dfa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc104bc0 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e0b96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e0b96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58104/0 (socket says 172.21.15.89:58104) 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e0b96700 1 -- 172.21.15.89:0/3322814828 learned_addr learned my addr 172.21.15.89:0/3322814828 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e0b96700 1 -- 172.21.15.89:0/3322814828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc19a280 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e0b96700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc199d40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f36d0003e40 tx=0x7f36d00119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36d001a430 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f36d001aa30 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36dc19a570 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.224+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36d0024960 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.225+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36dc19d200 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.226+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f36d0022020 con 0x7f36dc107470 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.226+0000 7f36d9ffb700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:18.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.226+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 --> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36cc03bd50 con 0x7f36cc039180 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.226+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f36d004d180 con 0x7f36dc107470 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.226+0000 7f36dbfff700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 msgr2=0x7f36cc03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3099145381 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.226+0000 7f36dbfff700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.427+0000 7f36dbfff700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 msgr2=0x7f36cc03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3099145381 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.427+0000 7f36dbfff700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.828+0000 7f36dbfff700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 msgr2=0x7f36cc03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3099145381 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:12.828+0000 7f36dbfff700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:13.629+0000 7f36dbfff700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 msgr2=0x7f36cc03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3099145381 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:13.629+0000 7f36dbfff700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:15.230+0000 7f36dbfff700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 msgr2=0x7f36cc03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3099145381 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:15.230+0000 7f36dbfff700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:16.699+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 6) v1 ==== 46061+0+0 (secure 0 0 0) 0x7f36d002d3f0 con 0x7f36dc107470 2024-03-21T13:06:18.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:16.699+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 msgr2=0x7f36cc03b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:16.699+0000 7f36d9ffb700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.700+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 7) v1 ==== 46188+0+0 (secure 0 0 0) 0x7f36d004e750 con 0x7f36dc107470 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.701+0000 7f36d9ffb700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f36cc03ca70 0x7f36cc03ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.701+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36d0001d30 con 0x7f36cc03ca70 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.701+0000 7f36dbfff700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f36cc03ca70 0x7f36cc03ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.702+0000 7f36dbfff700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f36cc03ca70 0x7f36cc03ee60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f36c8003a10 tx=0x7f36c80092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.702+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f36d0001d30 con 0x7f36cc03ca70 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.708+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f36dc0637c0 con 0x7f36cc03ca70 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36d9ffb700 1 -- 172.21.15.89:0/3322814828 <== mgr.14120 v2:172.21.15.89:6800/1022748022 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f36dc0637c0 con 0x7f36cc03ca70 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f36cc03ca70 msgr2=0x7f36cc03ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f36cc03ca70 0x7f36cc03ee60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f36c8003a10 tx=0x7f36c80092b0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 msgr2=0x7f36dc199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc199d40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f36d0003e40 tx=0x7f36d00119c0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 shutdown_connections 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f36cc03ca70 0x7f36cc03ee60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36dc107470 0x7f36dc199d40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 --2- 172.21.15.89:0/3322814828 >> [v2:172.21.15.89:6800/3099145381,v1:172.21.15.89:6801/3099145381] conn(0x7f36cc039180 0x7f36cc03b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 >> 172.21.15.89:0/3322814828 conn(0x7f36dc0ffa90 msgr2=0x7f36dc1006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 shutdown_connections 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:17.709+0000 7f36e2dfa700 1 -- 172.21.15.89:0/3322814828 wait complete. 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 5 is available 2024-03-21T13:06:18.255 INFO:teuthology.orchestra.run.smithi089.stdout:Setting orchestrator backend to cephadm... 2024-03-21T13:06:19.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:18 smithi089 ceph-mon[30095]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T13:06:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:18 smithi089 ceph-mon[30095]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T13:06:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:18 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb6e44b700 1 Processor -- start 2024-03-21T13:06:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb6e44b700 1 -- start start 2024-03-21T13:06:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb6e44b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb681051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb6e44b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb681057a0 con 0x7feb68104db0 2024-03-21T13:06:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb67fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb681051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb67fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb681051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43114/0 (socket says 172.21.15.89:43114) 2024-03-21T13:06:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb67fff700 1 -- 172.21.15.89:0/2731496230 learned_addr learned my addr 172.21.15.89:0/2731496230 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb67fff700 1 -- 172.21.15.89:0/2731496230 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68105fb0 con 0x7feb68104db0 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb67fff700 1 --2- 172.21.15.89:0/2731496230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb681051d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feb580098a0 tx=0x7feb58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb34096e3e16699 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb66ffd700 1 -- 172.21.15.89:0/2731496230 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb58004160 con 0x7feb68104db0 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.709+0000 7feb66ffd700 1 -- 172.21.15.89:0/2731496230 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7feb58004300 con 0x7feb68104db0 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb66ffd700 1 -- 172.21.15.89:0/2731496230 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb58011940 con 0x7feb68104db0 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb6e44b700 1 -- 172.21.15.89:0/2731496230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 msgr2=0x7feb681051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb6e44b700 1 --2- 172.21.15.89:0/2731496230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb681051d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feb580098a0 tx=0x7feb58009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb6e44b700 1 -- 172.21.15.89:0/2731496230 shutdown_connections 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb6e44b700 1 --2- 172.21.15.89:0/2731496230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb681051d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb6e44b700 1 -- 172.21.15.89:0/2731496230 >> 172.21.15.89:0/2731496230 conn(0x7feb68100370 msgr2=0x7feb68102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb6e44b700 1 -- 172.21.15.89:0/2731496230 shutdown_connections 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.710+0000 7feb6e44b700 1 -- 172.21.15.89:0/2731496230 wait complete. 2024-03-21T13:06:19.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb6e44b700 1 Processor -- start 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb6e44b700 1 -- start start 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb6e44b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb68198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb6e44b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb681091e0 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb67fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb68198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb67fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb68198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43122/0 (socket says 172.21.15.89:43122) 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb67fff700 1 -- 172.21.15.89:0/2931731960 learned_addr learned my addr 172.21.15.89:0/2931731960 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.711+0000 7feb67fff700 1 -- 172.21.15.89:0/2931731960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68199230 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.712+0000 7feb67fff700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb68198cf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb58011910 tx=0x7feb580119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.712+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb58003b20 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.712+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7feb5801a890 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.712+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb681994c0 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.712+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb580197f0 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.712+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6819c1a0 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.713+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 46188+0+0 (secure 0 0 0) 0x7feb58022020 con 0x7feb68104db0 2024-03-21T13:06:19.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.713+0000 7feb657fa700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7feb500390b0 0x7feb5003b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.713+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb5804df50 con 0x7feb68104db0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.713+0000 7feb677fe700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7feb500390b0 0x7feb5003b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.713+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb68109360 con 0x7feb68104db0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.713+0000 7feb677fe700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7feb500390b0 0x7feb5003b570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006fd0 tx=0x7feb5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.718+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb580172b0 con 0x7feb68104db0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.751+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7feb5802c720 con 0x7feb68104db0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.933+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7feb68101590 con 0x7feb500390b0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.942+0000 7feb657fa700 1 -- 172.21.15.89:0/2931731960 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feb68101590 con 0x7feb500390b0 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7feb500390b0 msgr2=0x7feb5003b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7feb500390b0 0x7feb5003b570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006fd0 tx=0x7feb5c006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 msgr2=0x7feb68198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb68198cf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb58011910 tx=0x7feb580119f0 comp rx=0 tx=0).stop 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 shutdown_connections 2024-03-21T13:06:19.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7feb500390b0 0x7feb5003b570 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:19.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 --2- 172.21.15.89:0/2931731960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb68104db0 0x7feb68198cf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:19.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 >> 172.21.15.89:0/2931731960 conn(0x7feb68100370 msgr2=0x7feb68100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:19.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 shutdown_connections 2024-03-21T13:06:19.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:18.944+0000 7feb6e44b700 1 -- 172.21.15.89:0/2931731960 wait complete. 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:18] ENGINE Bus STARTING 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:18] ENGINE Serving on http://172.21.15.89:8765 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:18] ENGINE Serving on https://172.21.15.89:7150 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:18] ENGINE Bus STARTED 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: mgrmap e8: y(active, since 2s) 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:19 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417b6fe700 1 Processor -- start 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417b6fe700 1 -- start start 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417b6fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f41741062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417b6fe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41741067f0 con 0x7f4174105e90 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417949a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f41741062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417949a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f41741062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43134/0 (socket says 172.21.15.89:43134) 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417949a700 1 -- 172.21.15.89:0/1913975037 learned_addr learned my addr 172.21.15.89:0/1913975037 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.973+0000 7f417949a700 1 -- 172.21.15.89:0/1913975037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4174107080 con 0x7f4174105e90 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417949a700 1 --2- 172.21.15.89:0/1913975037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f41741062b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41640098a0 tx=0x7f4164009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb387ef2874e9d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f416bfff700 1 -- 172.21.15.89:0/1913975037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4164004160 con 0x7f4174105e90 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f416bfff700 1 -- 172.21.15.89:0/1913975037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f4164004300 con 0x7f4174105e90 2024-03-21T13:06:20.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f416bfff700 1 -- 172.21.15.89:0/1913975037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4164011940 con 0x7f4174105e90 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417b6fe700 1 -- 172.21.15.89:0/1913975037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 msgr2=0x7f41741062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417b6fe700 1 --2- 172.21.15.89:0/1913975037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f41741062b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41640098a0 tx=0x7f4164009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417b6fe700 1 -- 172.21.15.89:0/1913975037 shutdown_connections 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417b6fe700 1 --2- 172.21.15.89:0/1913975037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f41741062b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417b6fe700 1 -- 172.21.15.89:0/1913975037 >> 172.21.15.89:0/1913975037 conn(0x7f4174101480 msgr2=0x7f41741038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417b6fe700 1 -- 172.21.15.89:0/1913975037 shutdown_connections 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.974+0000 7f417b6fe700 1 -- 172.21.15.89:0/1913975037 wait complete. 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.975+0000 7f417b6fe700 1 Processor -- start 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.975+0000 7f417b6fe700 1 -- start start 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.975+0000 7f417b6fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f417407b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.975+0000 7f417b6fe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4174106970 con 0x7f4174105e90 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.975+0000 7f417949a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f417407b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.975+0000 7f417949a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f417407b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43148/0 (socket says 172.21.15.89:43148) 2024-03-21T13:06:20.768 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.975+0000 7f417949a700 1 -- 172.21.15.89:0/368413296 learned_addr learned my addr 172.21.15.89:0/368413296 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.976+0000 7f417949a700 1 -- 172.21.15.89:0/368413296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417407b8c0 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.976+0000 7f417949a700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f417407b380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4164006ba0 tx=0x7f41640119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.976+0000 7f416a7fc700 1 -- 172.21.15.89:0/368413296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4164003930 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.976+0000 7f416a7fc700 1 -- 172.21.15.89:0/368413296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f4164003ad0 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.976+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4174079ab0 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.976+0000 7f416a7fc700 1 -- 172.21.15.89:0/368413296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4164019750 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.976+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4174079ff0 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.977+0000 7f416a7fc700 1 -- 172.21.15.89:0/368413296 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f4164019a10 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.977+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41740637c0 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.977+0000 7f416a7fc700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4160039130 0x7f416003b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.977+0000 7f416a7fc700 1 -- 172.21.15.89:0/368413296 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f416404d8a0 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.977+0000 7f4178c99700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4160039130 0x7f416003b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.977+0000 7f4178c99700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4160039130 0x7f416003b5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4170006fd0 tx=0x7f4170006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:19.982+0000 7f416a7fc700 1 -- 172.21.15.89:0/368413296 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41640172b0 con 0x7f4174105e90 2024-03-21T13:06:20.769 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.178+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f417407a7e0 con 0x7f4160039130 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.178+0000 7f416a7fc700 1 -- 172.21.15.89:0/368413296 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f417407a7e0 con 0x7f4160039130 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4160039130 msgr2=0x7f416003b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4160039130 0x7f416003b5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4170006fd0 tx=0x7f4170006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 msgr2=0x7f417407b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f417407b380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4164006ba0 tx=0x7f41640119c0 comp rx=0 tx=0).stop 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 shutdown_connections 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4160039130 0x7f416003b5f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 --2- 172.21.15.89:0/368413296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4174105e90 0x7f417407b380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 >> 172.21.15.89:0/368413296 conn(0x7f4174101480 msgr2=0x7f4174101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.180+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 shutdown_connections 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:20.181+0000 7f417b6fe700 1 -- 172.21.15.89:0/368413296 wait complete. 2024-03-21T13:06:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:Generating ssh key... 2024-03-21T13:06:21.023 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:20 smithi089 ceph-mon[30095]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:21.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: Generating public/private rsa key pair. 2024-03-21T13:06:21.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: Your identification has been saved in /tmp/tmptwpb8qee/key. 2024-03-21T13:06:21.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: Your public key has been saved in /tmp/tmptwpb8qee/key.pub. 2024-03-21T13:06:21.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: The key fingerprint is: 2024-03-21T13:06:21.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: SHA256:IOST3VSMi7H0poo3DLQdpG5xHJb1dibTIVA75LdChXc ceph-9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:06:21.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: The key's randomart image is: 2024-03-21T13:06:21.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: +---[RSA 3072]----+ 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | .ooo==+ | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | o=oo*o=o.E | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | ==oo=@o=. | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | + +oo++B . | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | o = . oS . | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | = . . . | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | . + . | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | . = | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: | . . | 2024-03-21T13:06:21.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:21 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: +----[SHA256]-----+ 2024-03-21T13:06:22.139 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.227+0000 7ff64fdac700 1 Processor -- start 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.227+0000 7ff64fdac700 1 -- start start 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.227+0000 7ff64fdac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff6481062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.227+0000 7ff64fdac700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6481067f0 con 0x7ff648105e90 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.227+0000 7ff64db48700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff6481062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.227+0000 7ff64db48700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff6481062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43160/0 (socket says 172.21.15.89:43160) 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.227+0000 7ff64db48700 1 -- 172.21.15.89:0/3494819208 learned_addr learned my addr 172.21.15.89:0/3494819208 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64db48700 1 -- 172.21.15.89:0/3494819208 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff648107080 con 0x7ff648105e90 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64db48700 1 --2- 172.21.15.89:0/3494819208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff6481062b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6440098a0 tx=0x7ff644009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13316eee580e71c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64cb46700 1 -- 172.21.15.89:0/3494819208 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff644004160 con 0x7ff648105e90 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64cb46700 1 -- 172.21.15.89:0/3494819208 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7ff644004300 con 0x7ff648105e90 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64cb46700 1 -- 172.21.15.89:0/3494819208 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff644011940 con 0x7ff648105e90 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64fdac700 1 -- 172.21.15.89:0/3494819208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 msgr2=0x7ff6481062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64fdac700 1 --2- 172.21.15.89:0/3494819208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff6481062b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6440098a0 tx=0x7ff644009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:22.140 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64fdac700 1 -- 172.21.15.89:0/3494819208 shutdown_connections 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64fdac700 1 --2- 172.21.15.89:0/3494819208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff6481062b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.228+0000 7ff64fdac700 1 -- 172.21.15.89:0/3494819208 >> 172.21.15.89:0/3494819208 conn(0x7ff648101480 msgr2=0x7ff6481038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64fdac700 1 -- 172.21.15.89:0/3494819208 shutdown_connections 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64fdac700 1 -- 172.21.15.89:0/3494819208 wait complete. 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64fdac700 1 Processor -- start 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64fdac700 1 -- start start 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64fdac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff648199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64fdac700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff648106970 con 0x7ff648105e90 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64db48700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff648199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64db48700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff648199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43174/0 (socket says 172.21.15.89:43174) 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.229+0000 7ff64db48700 1 -- 172.21.15.89:0/2353118822 learned_addr learned my addr 172.21.15.89:0/2353118822 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.230+0000 7ff64db48700 1 -- 172.21.15.89:0/2353118822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64819a360 con 0x7ff648105e90 2024-03-21T13:06:22.141 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.230+0000 7ff64db48700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff648199e20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff644011910 tx=0x7ff6440119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.230+0000 7ff63effd700 1 -- 172.21.15.89:0/2353118822 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff644003b20 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.230+0000 7ff63effd700 1 -- 172.21.15.89:0/2353118822 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7ff64401a890 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.230+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64819a5f0 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.230+0000 7ff63effd700 1 -- 172.21.15.89:0/2353118822 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6440197f0 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.230+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64819d2e0 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.231+0000 7ff63effd700 1 -- 172.21.15.89:0/2353118822 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7ff644022020 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.231+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6480637c0 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.231+0000 7ff63effd700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7ff6340391d0 0x7ff63403b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.231+0000 7ff63effd700 1 -- 172.21.15.89:0/2353118822 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff64404d360 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.231+0000 7ff64d347700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7ff6340391d0 0x7ff63403b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.231+0000 7ff64d347700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7ff6340391d0 0x7ff63403b690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff638006fd0 tx=0x7ff638006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.236+0000 7ff63effd700 1 -- 172.21.15.89:0/2353118822 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff64401ad90 con 0x7ff648105e90 2024-03-21T13:06:22.142 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.431+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6481026f0 con 0x7ff6340391d0 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.585+0000 7ff63effd700 1 -- 172.21.15.89:0/2353118822 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff6481026f0 con 0x7ff6340391d0 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7ff6340391d0 msgr2=0x7ff63403b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7ff6340391d0 0x7ff63403b690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff638006fd0 tx=0x7ff638006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 msgr2=0x7ff648199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff648199e20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff644011910 tx=0x7ff6440119f0 comp rx=0 tx=0).stop 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 shutdown_connections 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7ff6340391d0 0x7ff63403b690 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 --2- 172.21.15.89:0/2353118822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff648105e90 0x7ff648199e20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 >> 172.21.15.89:0/2353118822 conn(0x7ff648101480 msgr2=0x7ff648101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 shutdown_connections 2024-03-21T13:06:22.143 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:21.587+0000 7ff64fdac700 1 -- 172.21.15.89:0/2353118822 wait complete. 2024-03-21T13:06:23.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:22 smithi089 ceph-mon[30095]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:23.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:22 smithi089 ceph-mon[30095]: Generating ssh key... 2024-03-21T13:06:23.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:22 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:23.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:22 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:23.346 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8TNS8VfTI9BYnZ8hnyEpHI3TLDktC+SiSLTLNMl0eyJxqVjs6Fsla8tWmkCQ8F/ITbDSeU8x0WwUBRvgfu9Kjm9yOJV41eO+RyRM4ugwas9Sr+uwodvw/5wq+tQzaY4daviCjB38NCKrtsngWXiLpCenngHNWdjzVkp+1BKgrVBoTtAccj036hjt2O2kSZbQxkIHeja5GYdoOfod2hBppzYFMvCQK/GOdI6zUV0/UR0RetW3r1qF0zKZ9HMwMxMXRqpgpetoxI906lFjhumkaOj7bL3ybdcm5pxoXS9RvFIhpBKxO0+FmKBaaZXBuG7v1b/lBCZzR5yObtZwolZ8LP+Y4+FkHBfEEFD8TsHkSlkCFO6JKNXIcWSrRvRuB5x3nlMwm4hpC3QW3nrLwol5p75yEZSaanGlgHJpfWMRkLRjql9LAsuMbRO1k+nAsWDN6J/XxECEVN76TO+oBPxeOib//vJtLVSDSeWnEFSfBSWexHS6OFumrx1GOSu+Ow8= ceph-9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:06:23.346 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.584+0000 7f4d23964700 1 Processor -- start 2024-03-21T13:06:23.346 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.584+0000 7f4d23964700 1 -- start start 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.584+0000 7f4d23964700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c108d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.584+0000 7f4d23964700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0796a0 con 0x7f4d1c106960 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.584+0000 7f4d21700700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c108d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.585+0000 7f4d21700700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c108d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43184/0 (socket says 172.21.15.89:43184) 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.585+0000 7f4d21700700 1 -- 172.21.15.89:0/3156318468 learned_addr learned my addr 172.21.15.89:0/3156318468 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.585+0000 7f4d21700700 1 -- 172.21.15.89:0/3156318468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c109290 con 0x7f4d1c106960 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.585+0000 7f4d21700700 1 --2- 172.21.15.89:0/3156318468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c108d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d14009e80 tx=0x7f4d1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e2ea5129807898 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.585+0000 7f4d1bfff700 1 -- 172.21.15.89:0/3156318468 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d14017070 con 0x7f4d1c106960 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.585+0000 7f4d1bfff700 1 -- 172.21.15.89:0/3156318468 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f4d1400b8c0 con 0x7f4d1c106960 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.585+0000 7f4d1bfff700 1 -- 172.21.15.89:0/3156318468 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d14004370 con 0x7f4d1c106960 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 -- 172.21.15.89:0/3156318468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 msgr2=0x7f4d1c108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 --2- 172.21.15.89:0/3156318468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c108d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d14009e80 tx=0x7f4d1400b330 comp rx=0 tx=0).stop 2024-03-21T13:06:23.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 -- 172.21.15.89:0/3156318468 shutdown_connections 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 --2- 172.21.15.89:0/3156318468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c108d50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 -- 172.21.15.89:0/3156318468 >> 172.21.15.89:0/3156318468 conn(0x7f4d1c100330 msgr2=0x7f4d1c1027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 -- 172.21.15.89:0/3156318468 shutdown_connections 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 -- 172.21.15.89:0/3156318468 wait complete. 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 Processor -- start 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 -- start start 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c1061e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.586+0000 7f4d23964700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c10a120 con 0x7f4d1c106960 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d21700700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c1061e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d21700700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c1061e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43198/0 (socket says 172.21.15.89:43198) 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d21700700 1 -- 172.21.15.89:0/1495631537 learned_addr learned my addr 172.21.15.89:0/1495631537 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d21700700 1 -- 172.21.15.89:0/1495631537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c106720 con 0x7f4d1c106960 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d21700700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c1061e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d140072a0 tx=0x7f4d14003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:23.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d1a7fc700 1 -- 172.21.15.89:0/1495631537 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d14021020 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d1a7fc700 1 -- 172.21.15.89:0/1495631537 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f4d1400bde0 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c104910 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d1a7fc700 1 -- 172.21.15.89:0/1495631537 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d14017040 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.587+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c104e50 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.588+0000 7f4d1a7fc700 1 -- 172.21.15.89:0/1495631537 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f4d14024020 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.588+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1c0637c0 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.588+0000 7f4d1a7fc700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4d04039180 0x7f4d0403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.588+0000 7f4d20eff700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4d04039180 0x7f4d0403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.588+0000 7f4d1a7fc700 1 -- 172.21.15.89:0/1495631537 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4d1400f050 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.589+0000 7f4d20eff700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4d04039180 0x7f4d0403b640 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d10006fd0 tx=0x7f4d10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.593+0000 7f4d1a7fc700 1 -- 172.21.15.89:0/1495631537 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d1402b020 con 0x7f4d1c106960 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.790+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d1c105510 con 0x7f4d04039180 2024-03-21T13:06:23.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.791+0000 7f4d1a7fc700 1 -- 172.21.15.89:0/1495631537 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f4d1c105510 con 0x7f4d04039180 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4d04039180 msgr2=0x7f4d0403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4d04039180 0x7f4d0403b640 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d10006fd0 tx=0x7f4d10006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 msgr2=0x7f4d1c1061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c1061e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d140072a0 tx=0x7f4d14003d00 comp rx=0 tx=0).stop 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 shutdown_connections 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4d04039180 0x7f4d0403b640 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 --2- 172.21.15.89:0/1495631537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c106960 0x7f4d1c1061e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 >> 172.21.15.89:0/1495631537 conn(0x7f4d1c100330 msgr2=0x7f4d1c100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 shutdown_connections 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:22.793+0000 7f4d23964700 1 -- 172.21.15.89:0/1495631537 wait complete. 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:Adding key to root@localhost authorized_keys... 2024-03-21T13:06:23.350 INFO:teuthology.orchestra.run.smithi089.stdout:Adding host smithi089... 2024-03-21T13:06:23.682 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:23 smithi089 ceph-mon[30095]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:25.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:25 smithi089 ceph-mon[30095]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:25.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:25 smithi089 ceph-mon[30095]: Deploying cephadm binary to smithi089 2024-03-21T13:06:28.795 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Added host 'smithi089' with addr '172.21.15.89' 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.785+0000 7f5b4d034700 1 Processor -- start 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.785+0000 7f5b4d034700 1 -- start start 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.785+0000 7f5b4d034700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b481051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.785+0000 7f5b4d034700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b481057a0 con 0x7f5b48104db0 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.785+0000 7f5b46d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b481051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.785+0000 7f5b46d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b481051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43200/0 (socket says 172.21.15.89:43200) 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.785+0000 7f5b46d9d700 1 -- 172.21.15.89:0/133112981 learned_addr learned my addr 172.21.15.89:0/133112981 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.786+0000 7f5b46d9d700 1 -- 172.21.15.89:0/133112981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b48105fb0 con 0x7f5b48104db0 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.786+0000 7f5b46d9d700 1 --2- 172.21.15.89:0/133112981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b481051d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5b380098a0 tx=0x7f5b38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f548ab8096c82261 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.786+0000 7f5b4659c700 1 -- 172.21.15.89:0/133112981 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b38004160 con 0x7f5b48104db0 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.786+0000 7f5b4659c700 1 -- 172.21.15.89:0/133112981 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f5b38004300 con 0x7f5b48104db0 2024-03-21T13:06:28.796 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.786+0000 7f5b4659c700 1 -- 172.21.15.89:0/133112981 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b38011940 con 0x7f5b48104db0 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 -- 172.21.15.89:0/133112981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 msgr2=0x7f5b481051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 --2- 172.21.15.89:0/133112981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b481051d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5b380098a0 tx=0x7f5b38009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 -- 172.21.15.89:0/133112981 shutdown_connections 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 --2- 172.21.15.89:0/133112981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b481051d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 -- 172.21.15.89:0/133112981 >> 172.21.15.89:0/133112981 conn(0x7f5b48100370 msgr2=0x7f5b48102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 -- 172.21.15.89:0/133112981 shutdown_connections 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 -- 172.21.15.89:0/133112981 wait complete. 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 Processor -- start 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.787+0000 7f5b4d034700 1 -- start start 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b4d034700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b480792c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b4d034700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b481091e0 con 0x7f5b48104db0 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b46d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b480792c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:28.797 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b46d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b480792c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43206/0 (socket says 172.21.15.89:43206) 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b46d9d700 1 -- 172.21.15.89:0/255163419 learned_addr learned my addr 172.21.15.89:0/255163419 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b46d9d700 1 -- 172.21.15.89:0/255163419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4819cab0 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b46d9d700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b480792c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5b38003e40 tx=0x7f5b380119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b44d99700 1 -- 172.21.15.89:0/255163419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b3801a430 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b44d99700 1 -- 172.21.15.89:0/255163419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f5b3801aa30 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4819cc80 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b44d99700 1 -- 172.21.15.89:0/255163419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b38024960 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.788+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4819d0a0 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.789+0000 7f5b44d99700 1 -- 172.21.15.89:0/255163419 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f5b38022020 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.789+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b48109360 con 0x7f5b48104db0 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.789+0000 7f5b44d99700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f5b34039130 0x7f5b3403b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:28.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.789+0000 7f5b44d99700 1 -- 172.21.15.89:0/255163419 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5b3804d980 con 0x7f5b48104db0 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.789+0000 7f5b3ffff700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f5b34039130 0x7f5b3403b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.790+0000 7f5b3ffff700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f5b34039130 0x7f5b3403b5f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5b30006fd0 tx=0x7f5b30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.794+0000 7f5b44d99700 1 -- 172.21.15.89:0/255163419 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b3800b3e0 con 0x7f5b48104db0 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:23.993+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b481015c0 con 0x7f5b34039130 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.161+0000 7f5b44d99700 1 -- 172.21.15.89:0/255163419 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f5b481015c0 con 0x7f5b34039130 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f5b34039130 msgr2=0x7f5b3403b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f5b34039130 0x7f5b3403b5f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5b30006fd0 tx=0x7f5b30006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 msgr2=0x7f5b480792c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b480792c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5b38003e40 tx=0x7f5b380119c0 comp rx=0 tx=0).stop 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 shutdown_connections 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f5b34039130 0x7f5b3403b5f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:28.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 --2- 172.21.15.89:0/255163419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b48104db0 0x7f5b480792c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:28.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 >> 172.21.15.89:0/255163419 conn(0x7f5b48100370 msgr2=0x7f5b48100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:28.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 shutdown_connections 2024-03-21T13:06:28.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:28.164+0000 7f5b4d034700 1 -- 172.21.15.89:0/255163419 wait complete. 2024-03-21T13:06:28.800 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying unmanaged mon service... 2024-03-21T13:06:29.479 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:29 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:29.479 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:29 smithi089 ceph-mon[30095]: Added host smithi089 2024-03-21T13:06:29.480 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:29 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b79e30700 1 Processor -- start 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b79e30700 1 -- start start 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b79e30700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74108d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b79e30700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b74078df0 con 0x7f8b74106920 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b737fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74108d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b737fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74108d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41636/0 (socket says 172.21.15.89:41636) 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b737fe700 1 -- 172.21.15.89:0/94752890 learned_addr learned my addr 172.21.15.89:0/94752890 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b737fe700 1 -- 172.21.15.89:0/94752890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b74109250 con 0x7f8b74106920 2024-03-21T13:06:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b737fe700 1 --2- 172.21.15.89:0/94752890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74108d10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8b640098a0 tx=0x7f8b64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be337611508bb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.262+0000 7f8b727fc700 1 -- 172.21.15.89:0/94752890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b64004160 con 0x7f8b74106920 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b727fc700 1 -- 172.21.15.89:0/94752890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f8b64004300 con 0x7f8b74106920 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b727fc700 1 -- 172.21.15.89:0/94752890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b64011940 con 0x7f8b74106920 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b79e30700 1 -- 172.21.15.89:0/94752890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 msgr2=0x7f8b74108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b79e30700 1 --2- 172.21.15.89:0/94752890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74108d10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8b640098a0 tx=0x7f8b64009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b79e30700 1 -- 172.21.15.89:0/94752890 shutdown_connections 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b79e30700 1 --2- 172.21.15.89:0/94752890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74108d10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b79e30700 1 -- 172.21.15.89:0/94752890 >> 172.21.15.89:0/94752890 conn(0x7f8b74100310 msgr2=0x7f8b74102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b79e30700 1 -- 172.21.15.89:0/94752890 shutdown_connections 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.263+0000 7f8b79e30700 1 -- 172.21.15.89:0/94752890 wait complete. 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b79e30700 1 Processor -- start 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b79e30700 1 -- start start 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b79e30700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:30.039 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b79e30700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7410a150 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b737fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b737fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41644/0 (socket says 172.21.15.89:41644) 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b737fe700 1 -- 172.21.15.89:0/3148532939 learned_addr learned my addr 172.21.15.89:0/3148532939 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b737fe700 1 -- 172.21.15.89:0/3148532939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7419a220 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.264+0000 7f8b737fe700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74199ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b64003e40 tx=0x7f8b640119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.265+0000 7f8b70ff9700 1 -- 172.21.15.89:0/3148532939 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b64003b20 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.265+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7419a4b0 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.265+0000 7f8b70ff9700 1 -- 172.21.15.89:0/3148532939 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f8b6401a890 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.265+0000 7f8b70ff9700 1 -- 172.21.15.89:0/3148532939 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b64024890 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.265+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7419d190 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.265+0000 7f8b70ff9700 1 -- 172.21.15.89:0/3148532939 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f8b64022020 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.265+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b740637c0 con 0x7f8b74106920 2024-03-21T13:06:30.040 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.266+0000 7f8b70ff9700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8b5c039180 0x7f8b5c03b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.266+0000 7f8b70ff9700 1 -- 172.21.15.89:0/3148532939 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8b6404d590 con 0x7f8b74106920 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.266+0000 7f8b72ffd700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8b5c039180 0x7f8b5c03b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.266+0000 7f8b72ffd700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8b5c039180 0x7f8b5c03b640 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8b68006fd0 tx=0x7f8b68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.271+0000 7f8b70ff9700 1 -- 172.21.15.89:0/3148532939 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8b6401ad90 con 0x7f8b74106920 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.478+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8b74101530 con 0x7f8b5c039180 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.485+0000 7f8b70ff9700 1 -- 172.21.15.89:0/3148532939 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8b74101530 con 0x7f8b5c039180 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8b5c039180 msgr2=0x7f8b5c03b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8b5c039180 0x7f8b5c03b640 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8b68006fd0 tx=0x7f8b68006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 msgr2=0x7f8b74199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74199ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b64003e40 tx=0x7f8b640119c0 comp rx=0 tx=0).stop 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 shutdown_connections 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8b5c039180 0x7f8b5c03b640 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 --2- 172.21.15.89:0/3148532939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b74106920 0x7f8b74199ce0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:30.041 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 >> 172.21.15.89:0/3148532939 conn(0x7f8b74100310 msgr2=0x7f8b74100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:30.042 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 shutdown_connections 2024-03-21T13:06:30.042 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:29.488+0000 7f8b79e30700 1 -- 172.21.15.89:0/3148532939 wait complete. 2024-03-21T13:06:30.042 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying unmanaged mgr service... 2024-03-21T13:06:30.785 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:30 smithi089 ceph-mon[30095]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:30.785 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:30 smithi089 ceph-mon[30095]: Saving service mon spec with placement count:5 2024-03-21T13:06:30.785 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:30 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:31.982 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe781ded700 1 Processor -- start 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe781ded700 1 -- start start 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe781ded700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c104bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe781ded700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c079640 con 0x7fe77c104790 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe77b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c104bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe77b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c104bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41648/0 (socket says 172.21.15.89:41648) 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe77b7fe700 1 -- 172.21.15.89:0/1361599250 learned_addr learned my addr 172.21.15.89:0/1361599250 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.720+0000 7fe77b7fe700 1 -- 172.21.15.89:0/1361599250 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c1050f0 con 0x7fe77c104790 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.721+0000 7fe77b7fe700 1 --2- 172.21.15.89:0/1361599250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c104bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0098a0 tx=0x7fe76c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1748e0b507f985 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.721+0000 7fe77a7fc700 1 -- 172.21.15.89:0/1361599250 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe76c004160 con 0x7fe77c104790 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.721+0000 7fe77a7fc700 1 -- 172.21.15.89:0/1361599250 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fe76c004300 con 0x7fe77c104790 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.721+0000 7fe77a7fc700 1 -- 172.21.15.89:0/1361599250 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe76c011940 con 0x7fe77c104790 2024-03-21T13:06:31.983 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 -- 172.21.15.89:0/1361599250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 msgr2=0x7fe77c104bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 --2- 172.21.15.89:0/1361599250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c104bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0098a0 tx=0x7fe76c009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 -- 172.21.15.89:0/1361599250 shutdown_connections 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 --2- 172.21.15.89:0/1361599250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c104bb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 -- 172.21.15.89:0/1361599250 >> 172.21.15.89:0/1361599250 conn(0x7fe77c100370 msgr2=0x7fe77c102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 -- 172.21.15.89:0/1361599250 shutdown_connections 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 -- 172.21.15.89:0/1361599250 wait complete. 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.722+0000 7fe781ded700 1 Processor -- start 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe781ded700 1 -- start start 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe781ded700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c19bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe781ded700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c106060 con 0x7fe77c104790 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe77b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c19bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe77b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c19bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41658/0 (socket says 172.21.15.89:41658) 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe77b7fe700 1 -- 172.21.15.89:0/3618739840 learned_addr learned my addr 172.21.15.89:0/3618739840 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe77b7fe700 1 -- 172.21.15.89:0/3618739840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c19c4d0 con 0x7fe77c104790 2024-03-21T13:06:31.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe77b7fe700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c19bf90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009290 tx=0x7fe76c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe778ff9700 1 -- 172.21.15.89:0/3618739840 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe76c003930 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77c19c760 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe778ff9700 1 -- 172.21.15.89:0/3618739840 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fe76c003ad0 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.723+0000 7fe778ff9700 1 -- 172.21.15.89:0/3618739840 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe76c019720 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.724+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77c19f450 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.724+0000 7fe778ff9700 1 -- 172.21.15.89:0/3618739840 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7fe76c019a20 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.725+0000 7fe778ff9700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7fe764039130 0x7fe76403b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.725+0000 7fe778ff9700 1 -- 172.21.15.89:0/3618739840 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe76c04d900 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.725+0000 7fe77affd700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7fe764039130 0x7fe76403b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.725+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77c0637c0 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.725+0000 7fe77affd700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7fe764039130 0x7fe76403b5f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe770006fd0 tx=0x7fe770006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.730+0000 7fe778ff9700 1 -- 172.21.15.89:0/3618739840 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe76c020240 con 0x7fe77c104790 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.958+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe77c0625d0 con 0x7fe764039130 2024-03-21T13:06:31.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.965+0000 7fe778ff9700 1 -- 172.21.15.89:0/3618739840 <== mgr.14120 v2:172.21.15.89:6800/1022748022 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe77c0625d0 con 0x7fe764039130 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7fe764039130 msgr2=0x7fe76403b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7fe764039130 0x7fe76403b5f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe770006fd0 tx=0x7fe770006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 msgr2=0x7fe77c19bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c19bf90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009290 tx=0x7fe76c0119c0 comp rx=0 tx=0).stop 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 shutdown_connections 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7fe764039130 0x7fe76403b5f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 --2- 172.21.15.89:0/3618739840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe77c104790 0x7fe77c19bf90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 >> 172.21.15.89:0/3618739840 conn(0x7fe77c100370 msgr2=0x7fe77c192bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 shutdown_connections 2024-03-21T13:06:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:30.968+0000 7fe781ded700 1 -- 172.21.15.89:0/3618739840 wait complete. 2024-03-21T13:06:32.220 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:31 smithi089 ceph-mon[30095]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:32.220 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:31 smithi089 ceph-mon[30095]: Saving service mgr spec with placement count:2 2024-03-21T13:06:32.220 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:31 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:32.220 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:31 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:33.263 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.450+0000 7f8d5def5700 1 Processor -- start 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.450+0000 7f8d5def5700 1 -- start start 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.450+0000 7f8d5def5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d581078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.450+0000 7f8d5def5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d58107ea0 con 0x7f8d581074b0 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d581078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d581078d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41660/0 (socket says 172.21.15.89:41660) 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d577fe700 1 -- 172.21.15.89:0/1202890067 learned_addr learned my addr 172.21.15.89:0/1202890067 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d577fe700 1 -- 172.21.15.89:0/1202890067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d58108020 con 0x7f8d581074b0 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d577fe700 1 --2- 172.21.15.89:0/1202890067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d581078d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0098a0 tx=0x7f8d4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e751224b63d88e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d567fc700 1 -- 172.21.15.89:0/1202890067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d4c004160 con 0x7f8d581074b0 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d567fc700 1 -- 172.21.15.89:0/1202890067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8d4c004300 con 0x7f8d581074b0 2024-03-21T13:06:33.264 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.451+0000 7f8d567fc700 1 -- 172.21.15.89:0/1202890067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d4c0118f0 con 0x7f8d581074b0 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.452+0000 7f8d5def5700 1 -- 172.21.15.89:0/1202890067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 msgr2=0x7f8d581078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.452+0000 7f8d5def5700 1 --2- 172.21.15.89:0/1202890067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d581078d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0098a0 tx=0x7f8d4c009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.452+0000 7f8d5def5700 1 -- 172.21.15.89:0/1202890067 shutdown_connections 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.452+0000 7f8d5def5700 1 --2- 172.21.15.89:0/1202890067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d581078d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.452+0000 7f8d5def5700 1 -- 172.21.15.89:0/1202890067 >> 172.21.15.89:0/1202890067 conn(0x7f8d580ffa90 msgr2=0x7f8d58101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.452+0000 7f8d5def5700 1 -- 172.21.15.89:0/1202890067 shutdown_connections 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.452+0000 7f8d5def5700 1 -- 172.21.15.89:0/1202890067 wait complete. 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d5def5700 1 Processor -- start 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d5def5700 1 -- start start 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d5def5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d58199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d5def5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d58104cb0 con 0x7f8d581074b0 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d58199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d58199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41668/0 (socket says 172.21.15.89:41668) 2024-03-21T13:06:33.265 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d577fe700 1 -- 172.21.15.89:0/181863703 learned_addr learned my addr 172.21.15.89:0/181863703 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d577fe700 1 -- 172.21.15.89:0/181863703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5819a1e0 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.453+0000 7f8d577fe700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d58199ca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c003f60 tx=0x7f8d4c011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.454+0000 7f8d54ff9700 1 -- 172.21.15.89:0/181863703 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d4c0038b0 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.454+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5819a4d0 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.454+0000 7f8d54ff9700 1 -- 172.21.15.89:0/181863703 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8d4c003a50 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.454+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5819d160 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.454+0000 7f8d54ff9700 1 -- 172.21.15.89:0/181863703 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d4c024410 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.455+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d580637c0 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.455+0000 7f8d54ff9700 1 -- 172.21.15.89:0/181863703 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f8d4c022020 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.455+0000 7f8d54ff9700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8d40039180 0x7f8d4003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.455+0000 7f8d54ff9700 1 -- 172.21.15.89:0/181863703 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8d4c04d0d0 con 0x7f8d581074b0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.455+0000 7f8d56ffd700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8d40039180 0x7f8d4003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:33.266 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.455+0000 7f8d56ffd700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8d40039180 0x7f8d4003b640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8d48006fd0 tx=0x7f8d48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.464+0000 7f8d54ff9700 1 -- 172.21.15.89:0/181863703 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d4c00b3e0 con 0x7f8d581074b0 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.662+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f8d5819d5f0 con 0x7f8d581074b0 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.665+0000 7f8d54ff9700 1 -- 172.21.15.89:0/181863703 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8d4c019de0 con 0x7f8d581074b0 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8d40039180 msgr2=0x7f8d4003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8d40039180 0x7f8d4003b640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8d48006fd0 tx=0x7f8d48006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 msgr2=0x7f8d58199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d58199ca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c003f60 tx=0x7f8d4c011ef0 comp rx=0 tx=0).stop 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 shutdown_connections 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f8d40039180 0x7f8d4003b640 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 --2- 172.21.15.89:0/181863703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d581074b0 0x7f8d58199ca0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 >> 172.21.15.89:0/181863703 conn(0x7f8d580ffa90 msgr2=0x7f8d581005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 shutdown_connections 2024-03-21T13:06:33.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:32.668+0000 7f8d5def5700 1 -- 172.21.15.89:0/181863703 wait complete. 2024-03-21T13:06:33.506 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:33 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:33.507 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/181863703' entity='client.admin' 2024-03-21T13:06:34.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.726+0000 7f147201e700 1 Processor -- start 2024-03-21T13:06:34.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.726+0000 7f147201e700 1 -- start start 2024-03-21T13:06:34.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f147201e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c105190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:34.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f147201e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c105760 con 0x7f146c104d70 2024-03-21T13:06:34.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:34.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41670/0 (socket says 172.21.15.89:41670) 2024-03-21T13:06:34.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146b7fe700 1 -- 172.21.15.89:0/3966548523 learned_addr learned my addr 172.21.15.89:0/3966548523 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146b7fe700 1 -- 172.21.15.89:0/3966548523 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c105f70 con 0x7f146c104d70 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146b7fe700 1 --2- 172.21.15.89:0/3966548523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c105190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f145c0098a0 tx=0x7f145c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1132814950c24f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146a7fc700 1 -- 172.21.15.89:0/3966548523 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f145c004160 con 0x7f146c104d70 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146a7fc700 1 -- 172.21.15.89:0/3966548523 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f145c004300 con 0x7f146c104d70 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.727+0000 7f146a7fc700 1 -- 172.21.15.89:0/3966548523 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f145c0118f0 con 0x7f146c104d70 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.728+0000 7f147201e700 1 -- 172.21.15.89:0/3966548523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 msgr2=0x7f146c105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.728+0000 7f147201e700 1 --2- 172.21.15.89:0/3966548523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c105190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f145c0098a0 tx=0x7f145c009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:34.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.728+0000 7f147201e700 1 -- 172.21.15.89:0/3966548523 shutdown_connections 2024-03-21T13:06:34.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.728+0000 7f147201e700 1 --2- 172.21.15.89:0/3966548523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c105190 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:34.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.728+0000 7f147201e700 1 -- 172.21.15.89:0/3966548523 >> 172.21.15.89:0/3966548523 conn(0x7f146c100310 msgr2=0x7f146c102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:34.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.728+0000 7f147201e700 1 -- 172.21.15.89:0/3966548523 shutdown_connections 2024-03-21T13:06:34.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.728+0000 7f147201e700 1 -- 172.21.15.89:0/3966548523 wait complete. 2024-03-21T13:06:34.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.729+0000 7f147201e700 1 Processor -- start 2024-03-21T13:06:34.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.729+0000 7f147201e700 1 -- start start 2024-03-21T13:06:34.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.729+0000 7f147201e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c18fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:34.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.729+0000 7f147201e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c1091a0 con 0x7f146c104d70 2024-03-21T13:06:34.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.729+0000 7f146b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c18fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:34.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.729+0000 7f146b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c18fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41682/0 (socket says 172.21.15.89:41682) 2024-03-21T13:06:34.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.729+0000 7f146b7fe700 1 -- 172.21.15.89:0/3128523330 learned_addr learned my addr 172.21.15.89:0/3128523330 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:34.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.730+0000 7f146b7fe700 1 -- 172.21.15.89:0/3128523330 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c190370 con 0x7f146c104d70 2024-03-21T13:06:34.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.730+0000 7f146b7fe700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c18fe30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f145c003f60 tx=0x7f145c011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.730+0000 7f1468ff9700 1 -- 172.21.15.89:0/3128523330 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f145c0038b0 con 0x7f146c104d70 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.730+0000 7f1468ff9700 1 -- 172.21.15.89:0/3128523330 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f145c003a50 con 0x7f146c104d70 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.730+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146c18e560 con 0x7f146c104d70 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.730+0000 7f1468ff9700 1 -- 172.21.15.89:0/3128523330 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f145c024410 con 0x7f146c104d70 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.730+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c18eaa0 con 0x7f146c104d70 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.731+0000 7f1468ff9700 1 -- 172.21.15.89:0/3128523330 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f145c022020 con 0x7f146c104d70 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.731+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146c109320 con 0x7f146c104d70 2024-03-21T13:06:34.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.731+0000 7f1468ff9700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f1454039230 0x7f145403b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:34.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.731+0000 7f1468ff9700 1 -- 172.21.15.89:0/3128523330 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f145c04dec0 con 0x7f146c104d70 2024-03-21T13:06:34.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.732+0000 7f146affd700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f1454039230 0x7f145403b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:34.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.732+0000 7f146affd700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f1454039230 0x7f145403b6f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1460006fd0 tx=0x7f1460006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:34.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.736+0000 7f1468ff9700 1 -- 172.21.15.89:0/3128523330 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f145c0172b0 con 0x7f146c104d70 2024-03-21T13:06:34.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.942+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f146c193990 con 0x7f146c104d70 2024-03-21T13:06:34.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.945+0000 7f1468ff9700 1 -- 172.21.15.89:0/3128523330 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f145c00b5a0 con 0x7f146c104d70 2024-03-21T13:06:34.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f1454039230 msgr2=0x7f145403b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:34.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f1454039230 0x7f145403b6f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1460006fd0 tx=0x7f1460006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:34.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 msgr2=0x7f146c18fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:34.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c18fe30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f145c003f60 tx=0x7f145c011ef0 comp rx=0 tx=0).stop 2024-03-21T13:06:34.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 shutdown_connections 2024-03-21T13:06:34.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f1454039230 0x7f145403b6f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:34.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 --2- 172.21.15.89:0/3128523330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c104d70 0x7f146c18fe30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:34.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 >> 172.21.15.89:0/3128523330 conn(0x7f146c100310 msgr2=0x7f146c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:34.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 shutdown_connections 2024-03-21T13:06:34.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:33.947+0000 7f147201e700 1 -- 172.21.15.89:0/3128523330 wait complete. 2024-03-21T13:06:34.529 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling the dashboard module... 2024-03-21T13:06:35.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3128523330' entity='client.admin' 2024-03-21T13:06:36.255 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/330240921' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-21T13:06:36.255 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:35 smithi089 ceph-mon[30095]: from='mgr.14120 172.21.15.89:0/650304032' entity='mgr.y' 2024-03-21T13:06:36.256 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:36 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ignoring --setuser ceph since I am not root 2024-03-21T13:06:36.256 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:36 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ignoring --setgroup ceph since I am not root 2024-03-21T13:06:36.549 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.084+0000 7f432acc5700 1 Processor -- start 2024-03-21T13:06:36.549 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.084+0000 7f432acc5700 1 -- start start 2024-03-21T13:06:36.549 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.084+0000 7f432acc5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324105190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:36.549 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.084+0000 7f432acc5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4324105760 con 0x7f4324104d70 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f4328a61700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f4328a61700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41696/0 (socket says 172.21.15.89:41696) 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f4328a61700 1 -- 172.21.15.89:0/1940130979 learned_addr learned my addr 172.21.15.89:0/1940130979 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f4328a61700 1 -- 172.21.15.89:0/1940130979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4324105f70 con 0x7f4324104d70 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f4328a61700 1 --2- 172.21.15.89:0/1940130979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324105190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43180098a0 tx=0x7f4318009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef0d2cd6d7cdbc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f43237fe700 1 -- 172.21.15.89:0/1940130979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4318004160 con 0x7f4324104d70 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f43237fe700 1 -- 172.21.15.89:0/1940130979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4318004300 con 0x7f4324104d70 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.085+0000 7f43237fe700 1 -- 172.21.15.89:0/1940130979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43180118f0 con 0x7f4324104d70 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 -- 172.21.15.89:0/1940130979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 msgr2=0x7f4324105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 --2- 172.21.15.89:0/1940130979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324105190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43180098a0 tx=0x7f4318009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 -- 172.21.15.89:0/1940130979 shutdown_connections 2024-03-21T13:06:36.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 --2- 172.21.15.89:0/1940130979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324105190 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 -- 172.21.15.89:0/1940130979 >> 172.21.15.89:0/1940130979 conn(0x7f4324100310 msgr2=0x7f4324102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 -- 172.21.15.89:0/1940130979 shutdown_connections 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 -- 172.21.15.89:0/1940130979 wait complete. 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 Processor -- start 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 -- start start 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.086+0000 7f432acc5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4324077f50 con 0x7f4324104d70 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4328a61700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4328a61700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41698/0 (socket says 172.21.15.89:41698) 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4328a61700 1 -- 172.21.15.89:0/330240921 learned_addr learned my addr 172.21.15.89:0/330240921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4328a61700 1 -- 172.21.15.89:0/330240921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432419a230 con 0x7f4324104d70 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4328a61700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324199cf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4318003f60 tx=0x7f43180119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:36.551 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4318003a40 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f43180248b0 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432419a4c0 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431801a780 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.087+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432419d1a0 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.088+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f431801aa80 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.088+0000 7f4321ffb700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4314039130 0x7f431403b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.088+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f431804d770 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.088+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43240637c0 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.088+0000 7f4323fff700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4314039130 0x7f431403b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.089+0000 7f4323fff700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4314039130 0x7f431403b5f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4310006fd0 tx=0x7f4310006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.093+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f431804d450 con 0x7f4324104d70 2024-03-21T13:06:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.349+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f432402dad0 con 0x7f4324104d70 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.948+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f431801ad70 con 0x7f4324104d70 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.950+0000 7f4321ffb700 1 -- 172.21.15.89:0/330240921 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 9) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f431802d910 con 0x7f4324104d70 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4314039130 msgr2=0x7f431403b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4314039130 0x7f431403b5f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4310006fd0 tx=0x7f4310006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 msgr2=0x7f4324199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324199cf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4318003f60 tx=0x7f43180119b0 comp rx=0 tx=0).stop 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 shutdown_connections 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f4314039130 0x7f431403b5f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 --2- 172.21.15.89:0/330240921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4324104d70 0x7f4324199cf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 >> 172.21.15.89:0/330240921 conn(0x7f4324100310 msgr2=0x7f4324100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 shutdown_connections 2024-03-21T13:06:36.553 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:35.952+0000 7f432acc5700 1 -- 172.21.15.89:0/330240921 wait complete. 2024-03-21T13:06:36.818 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:36 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:36.531+0000 7f8949ffc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:06:36.818 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:36 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:36.630+0000 7f8949ffc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:06:37.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/330240921' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-21T13:06:37.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:36 smithi089 ceph-mon[30095]: mgrmap e9: y(active, since 19s) 2024-03-21T13:06:37.324 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:37.289+0000 7f8949ffc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:06:37.628 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:37.386+0000 7f8949ffc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:06:37.628 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:37.531+0000 7f8949ffc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:06:37.628 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:37.627+0000 7f8949ffc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f92a2593700 1 Processor -- start 2024-03-21T13:06:37.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f92a2593700 1 -- start start 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f92a2593700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c079b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f92a2593700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c07a120 con 0x7f929c07b6f0 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f929bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c079b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f929bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c079b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41728/0 (socket says 172.21.15.89:41728) 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f929bfff700 1 -- 172.21.15.89:0/757752691 learned_addr learned my addr 172.21.15.89:0/757752691 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f929bfff700 1 -- 172.21.15.89:0/757752691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c07a2a0 con 0x7f929c07b6f0 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.066+0000 7f929bfff700 1 --2- 172.21.15.89:0/757752691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c079b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f928c009ab0 tx=0x7f928c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40932c187a1c4fc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f929affd700 1 -- 172.21.15.89:0/757752691 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f928c004160 con 0x7f929c07b6f0 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f929affd700 1 -- 172.21.15.89:0/757752691 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f928c00bc60 con 0x7f929c07b6f0 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f929affd700 1 -- 172.21.15.89:0/757752691 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f928c003ac0 con 0x7f929c07b6f0 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f92a2593700 1 -- 172.21.15.89:0/757752691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 msgr2=0x7f929c079b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f92a2593700 1 --2- 172.21.15.89:0/757752691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c079b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f928c009ab0 tx=0x7f928c009dc0 comp rx=0 tx=0).stop 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f92a2593700 1 -- 172.21.15.89:0/757752691 shutdown_connections 2024-03-21T13:06:37.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f92a2593700 1 --2- 172.21.15.89:0/757752691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c079b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.067+0000 7f92a2593700 1 -- 172.21.15.89:0/757752691 >> 172.21.15.89:0/757752691 conn(0x7f929c101450 msgr2=0x7f929c1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.068+0000 7f92a2593700 1 -- 172.21.15.89:0/757752691 shutdown_connections 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.068+0000 7f92a2593700 1 -- 172.21.15.89:0/757752691 wait complete. 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.068+0000 7f92a2593700 1 Processor -- start 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.068+0000 7f92a2593700 1 -- start start 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.069+0000 7f92a2593700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.069+0000 7f92a2593700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c10c420 con 0x7f929c07b6f0 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.069+0000 7f929bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.069+0000 7f929bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41744/0 (socket says 172.21.15.89:41744) 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.069+0000 7f929bfff700 1 -- 172.21.15.89:0/1012760026 learned_addr learned my addr 172.21.15.89:0/1012760026 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.069+0000 7f929bfff700 1 -- 172.21.15.89:0/1012760026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c19c450 con 0x7f929c07b6f0 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.070+0000 7f929bfff700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c19bf10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f928c0037e0 tx=0x7f928c00bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.070+0000 7f92997fa700 1 -- 172.21.15.89:0/1012760026 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f928c003dc0 con 0x7f929c07b6f0 2024-03-21T13:06:37.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.070+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c19c6e0 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.070+0000 7f92997fa700 1 -- 172.21.15.89:0/1012760026 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f928c003f60 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.070+0000 7f92997fa700 1 -- 172.21.15.89:0/1012760026 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f928c011420 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.070+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c19f3d0 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.071+0000 7f92997fa700 1 -- 172.21.15.89:0/1012760026 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f928c0116e0 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.071+0000 7f92997fa700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 0x7f928403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.071+0000 7f92997fa700 1 -- 172.21.15.89:0/1012760026 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f928c054060 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.071+0000 7f929b7fe700 1 -- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 msgr2=0x7f928403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1022748022 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.071+0000 7f929b7fe700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 0x7f928403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.071+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c0637c0 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.077+0000 7f92997fa700 1 -- 172.21.15.89:0/1012760026 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f928c023c70 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.272+0000 7f929b7fe700 1 -- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 msgr2=0x7f928403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1022748022 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.272+0000 7f929b7fe700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 0x7f928403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.337+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f929c19f6e0 con 0x7f929c07b6f0 2024-03-21T13:06:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.337+0000 7f92997fa700 1 -- 172.21.15.89:0/1012760026 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f928c018a20 con 0x7f929c07b6f0 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 msgr2=0x7f928403b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 0x7f928403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 msgr2=0x7f929c19bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c19bf10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f928c0037e0 tx=0x7f928c00bac0 comp rx=0 tx=0).stop 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 shutdown_connections 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f9284039180 0x7f928403b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 --2- 172.21.15.89:0/1012760026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c07b6f0 0x7f929c19bf10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 >> 172.21.15.89:0/1012760026 conn(0x7f929c101450 msgr2=0x7f929c101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 shutdown_connections 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:37.339+0000 7f92a2593700 1 -- 172.21.15.89:0/1012760026 wait complete. 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2024-03-21T13:06:37.818 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 9... 2024-03-21T13:06:37.938 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:37.712+0000 7f8949ffc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:06:38.202 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1012760026' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T13:06:38.203 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:37.996+0000 7f8949ffc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:06:38.203 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:38.110+0000 7f8949ffc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:06:38.203 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:38.201+0000 7f8949ffc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:06:38.491 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:38.303+0000 7f8949ffc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:06:38.492 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:38.491+0000 7f8949ffc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:06:39.733 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:39 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:39.476+0000 7f8949ffc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:06:39.734 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:39 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:39.607+0000 7f8949ffc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:06:40.030 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:39 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:39.940+0000 7f8949ffc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:06:40.030 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:40 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:40.029+0000 7f8949ffc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:06:40.324 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:40 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:40.226+0000 7f8949ffc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:06:40.698 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:40 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:40.412+0000 7f8949ffc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:06:40.970 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:40 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:40.697+0000 7f8949ffc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:06:40.971 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:40 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:40.970+0000 7f8949ffc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:06:41.324 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:41 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:41.207+0000 7f8949ffc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:06:42.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:42.009+0000 7f8949ffc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:06:42.892 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:42.574+0000 7f8949ffc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:06:42.892 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:42.672+0000 7f8949ffc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:06:42.892 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:06:42.891+0000 7f8949ffc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: Active manager daemon y restarted 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: Activating manager daemon y 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: osdmap e3: 0 total, 0 up, 0 in 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: mgrmap e10: y(active, starting, since 0.00322939s) 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: Manager daemon y is now available 2024-03-21T13:06:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:44.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:06:44.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:06:44.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:43 smithi089 ceph-mon[30095]: mgrmap e11: y(active, since 1.00564s) 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.324+0000 7f6dcfe99700 1 Processor -- start 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.325+0000 7f6dcfe99700 1 -- start start 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.325+0000 7f6dcfe99700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8104930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.325+0000 7f6dcfe99700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc8104f00 con 0x7f6dc8104510 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.325+0000 7f6dcdc35700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8104930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.325+0000 7f6dcdc35700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8104930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43372/0 (socket says 172.21.15.89:43372) 2024-03-21T13:06:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.325+0000 7f6dcdc35700 1 -- 172.21.15.89:0/1468660081 learned_addr learned my addr 172.21.15.89:0/1468660081 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.325+0000 7f6dcdc35700 1 -- 172.21.15.89:0/1468660081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8105710 con 0x7f6dc8104510 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.326+0000 7f6dcdc35700 1 --2- 172.21.15.89:0/1468660081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8104930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6dc400aa40 tx=0x7f6dc400ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9ce7ca45f9049b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.326+0000 7f6dccc33700 1 -- 172.21.15.89:0/1468660081 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc400b040 con 0x7f6dc8104510 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.326+0000 7f6dccc33700 1 -- 172.21.15.89:0/1468660081 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6dc400c8f0 con 0x7f6dc8104510 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.326+0000 7f6dccc33700 1 -- 172.21.15.89:0/1468660081 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc4003a00 con 0x7f6dc8104510 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.326+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1468660081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 msgr2=0x7f6dc8104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.326+0000 7f6dcfe99700 1 --2- 172.21.15.89:0/1468660081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8104930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6dc400aa40 tx=0x7f6dc400ad50 comp rx=0 tx=0).stop 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.327+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1468660081 shutdown_connections 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.327+0000 7f6dcfe99700 1 --2- 172.21.15.89:0/1468660081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8104930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.327+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1468660081 >> 172.21.15.89:0/1468660081 conn(0x7f6dc80ffa90 msgr2=0x7f6dc8101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.327+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1468660081 shutdown_connections 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.327+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1468660081 wait complete. 2024-03-21T13:06:44.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.327+0000 7f6dcfe99700 1 Processor -- start 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.327+0000 7f6dcfe99700 1 -- start start 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcfe99700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcfe99700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc8105dc0 con 0x7f6dc8104510 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcdc35700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcdc35700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43384/0 (socket says 172.21.15.89:43384) 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcdc35700 1 -- 172.21.15.89:0/1836741167 learned_addr learned my addr 172.21.15.89:0/1836741167 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcdc35700 1 -- 172.21.15.89:0/1836741167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc819a300 con 0x7f6dc8104510 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcdc35700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8199dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4003750 tx=0x7f6dc4004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc400cd10 con 0x7f6dc8104510 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6dc4010a80 con 0x7f6dc8104510 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc819a590 con 0x7f6dc8104510 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.328+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc4018890 con 0x7f6dc8104510 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.329+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc819d280 con 0x7f6dc8104510 2024-03-21T13:06:44.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.330+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f6dc401f070 con 0x7f6dc8104510 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.330+0000 7f6dbeffd700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.330+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 --> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6db403bd50 con 0x7f6db4039180 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.330+0000 7f6dcd434700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 msgr2=0x7f6db403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1022748022 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.330+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6dc404d610 con 0x7f6dc8104510 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.330+0000 7f6dcd434700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.531+0000 7f6dcd434700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 msgr2=0x7f6db403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1022748022 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.531+0000 7f6dcd434700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.932+0000 7f6dcd434700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 msgr2=0x7f6db403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1022748022 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:38.932+0000 7f6dcd434700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:39.733+0000 7f6dcd434700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 msgr2=0x7f6db403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1022748022 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:39.733+0000 7f6dcd434700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:41.334+0000 7f6dcd434700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 msgr2=0x7f6db403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1022748022 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:41.334+0000 7f6dcd434700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-21T13:06:44.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:42.906+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 10) v1 ==== 46074+0+0 (secure 0 0 0) 0x7f6dc4010600 con 0x7f6dc8104510 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:42.906+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 msgr2=0x7f6db403b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:42.906+0000 7f6dbeffd700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.907+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 11) v1 ==== 46201+0+0 (secure 0 0 0) 0x7f6dc404ed20 con 0x7f6dc8104510 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.907+0000 7f6dbeffd700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f6db403cc00 0x7f6db403eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.908+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dc4046730 con 0x7f6db403cc00 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.908+0000 7f6dcd434700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f6db403cc00 0x7f6db403eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.909+0000 7f6dcd434700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f6db403cc00 0x7f6db403eff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6db8003a10 tx=0x7f6db80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.909+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f6dc4046730 con 0x7f6db403cc00 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6dc80637c0 con 0x7f6db403cc00 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dbeffd700 1 -- 172.21.15.89:0/1836741167 <== mgr.14152 v2:172.21.15.89:6800/4277445380 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f6dc80637c0 con 0x7f6db403cc00 2024-03-21T13:06:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f6db403cc00 msgr2=0x7f6db403eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f6db403cc00 0x7f6db403eff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6db8003a10 tx=0x7f6db80092b0 comp rx=0 tx=0).stop 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 msgr2=0x7f6dc8199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8199dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4003750 tx=0x7f6dc4004350 comp rx=0 tx=0).stop 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 shutdown_connections 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f6db403cc00 0x7f6db403eff0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc8104510 0x7f6dc8199dc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 --2- 172.21.15.89:0/1836741167 >> [v2:172.21.15.89:6800/1022748022,v1:172.21.15.89:6801/1022748022] conn(0x7f6db4039180 0x7f6db403b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 >> 172.21.15.89:0/1836741167 conn(0x7f6dc80ffa90 msgr2=0x7f6dc818e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 shutdown_connections 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:43.916+0000 7f6dcfe99700 1 -- 172.21.15.89:0/1836741167 wait complete. 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 9 is available 2024-03-21T13:06:44.704 INFO:teuthology.orchestra.run.smithi089.stdout:Generating a dashboard self-signed certificate... 2024-03-21T13:06:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:44 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:43] ENGINE Bus STARTING 2024-03-21T13:06:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:44 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:43] ENGINE Serving on http://172.21.15.89:8765 2024-03-21T13:06:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:44 smithi089 ceph-mon[30095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T13:06:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:44 smithi089 ceph-mon[30095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T13:06:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:44 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:43] ENGINE Serving on https://172.21.15.89:7150 2024-03-21T13:06:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:44 smithi089 ceph-mon[30095]: [21/Mar/2024:13:06:43] ENGINE Bus STARTED 2024-03-21T13:06:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:45 smithi089 ceph-mon[30095]: mgrmap e12: y(active, since 2s) 2024-03-21T13:06:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:46.363 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-21T13:06:46.363 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.497+0000 7f4879344700 1 Processor -- start 2024-03-21T13:06:46.363 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.497+0000 7f4879344700 1 -- start start 2024-03-21T13:06:46.363 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.497+0000 7f4879344700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f4874106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:46.363 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.497+0000 7f4879344700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4874106d60 con 0x7f4874106370 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4872ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f4874106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4872ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f4874106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43450/0 (socket says 172.21.15.89:43450) 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4872ffd700 1 -- 172.21.15.89:0/2443956853 learned_addr learned my addr 172.21.15.89:0/2443956853 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4872ffd700 1 -- 172.21.15.89:0/2443956853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4874104f80 con 0x7f4874106370 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4872ffd700 1 --2- 172.21.15.89:0/2443956853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f4874106790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48640098a0 tx=0x7f4864009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6b468cc81f582b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4871ffb700 1 -- 172.21.15.89:0/2443956853 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4864004160 con 0x7f4874106370 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4871ffb700 1 -- 172.21.15.89:0/2443956853 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4864004300 con 0x7f4874106370 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.498+0000 7f4871ffb700 1 -- 172.21.15.89:0/2443956853 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48640118f0 con 0x7f4874106370 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.499+0000 7f4879344700 1 -- 172.21.15.89:0/2443956853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 msgr2=0x7f4874106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.499+0000 7f4879344700 1 --2- 172.21.15.89:0/2443956853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f4874106790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48640098a0 tx=0x7f4864009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.499+0000 7f4879344700 1 -- 172.21.15.89:0/2443956853 shutdown_connections 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.499+0000 7f4879344700 1 --2- 172.21.15.89:0/2443956853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f4874106790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:46.364 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.499+0000 7f4879344700 1 -- 172.21.15.89:0/2443956853 >> 172.21.15.89:0/2443956853 conn(0x7f48740ffa90 msgr2=0x7f4874101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.499+0000 7f4879344700 1 -- 172.21.15.89:0/2443956853 shutdown_connections 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.499+0000 7f4879344700 1 -- 172.21.15.89:0/2443956853 wait complete. 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.500+0000 7f4879344700 1 Processor -- start 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.500+0000 7f4879344700 1 -- start start 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.500+0000 7f4879344700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f48741048f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.500+0000 7f4879344700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48741091c0 con 0x7f4874106370 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f4872ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f48741048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f4872ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f48741048f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43464/0 (socket says 172.21.15.89:43464) 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f4872ffd700 1 -- 172.21.15.89:0/685913090 learned_addr learned my addr 172.21.15.89:0/685913090 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f4872ffd700 1 -- 172.21.15.89:0/685913090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487419ca80 con 0x7f4874106370 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f4872ffd700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f48741048f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4864003f60 tx=0x7f4864011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f485bfff700 1 -- 172.21.15.89:0/685913090 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48640038b0 con 0x7f4874106370 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487419cc50 con 0x7f4874106370 2024-03-21T13:06:46.365 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f485bfff700 1 -- 172.21.15.89:0/685913090 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4864003a50 con 0x7f4874106370 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f485bfff700 1 -- 172.21.15.89:0/685913090 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4864024410 con 0x7f4874106370 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.501+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487419cff0 con 0x7f4874106370 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.502+0000 7f485bfff700 1 -- 172.21.15.89:0/685913090 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f4864022020 con 0x7f4874106370 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.502+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4874109340 con 0x7f4874106370 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.502+0000 7f485bfff700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f485c0391d0 0x7f485c03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.503+0000 7f485bfff700 1 -- 172.21.15.89:0/685913090 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f486404d130 con 0x7f4874106370 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.503+0000 7f48727fc700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f485c0391d0 0x7f485c03b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.503+0000 7f48727fc700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f485c0391d0 0x7f485c03b690 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4868006fd0 tx=0x7f4868006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.507+0000 7f485bfff700 1 -- 172.21.15.89:0/685913090 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f48640172b0 con 0x7f4874106370 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.713+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f4874100d00 con 0x7f485c0391d0 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.856+0000 7f485bfff700 1 -- 172.21.15.89:0/685913090 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f4874100d00 con 0x7f485c0391d0 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f485c0391d0 msgr2=0x7f485c03b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:46.366 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f485c0391d0 0x7f485c03b690 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4868006fd0 tx=0x7f4868006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 msgr2=0x7f48741048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f48741048f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4864003f60 tx=0x7f4864011ef0 comp rx=0 tx=0).stop 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 shutdown_connections 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f485c0391d0 0x7f485c03b690 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 --2- 172.21.15.89:0/685913090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4874106370 0x7f48741048f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.858+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 >> 172.21.15.89:0/685913090 conn(0x7f48740ffa90 msgr2=0x7f48741005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.859+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 shutdown_connections 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:45.859+0000 7f4879344700 1 -- 172.21.15.89:0/685913090 wait complete. 2024-03-21T13:06:46.367 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial admin user... 2024-03-21T13:06:47.051 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:46 smithi089 ceph-mon[30095]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:47.051 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:46 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:47.828 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$X.MB0OL5B3MP8nPpecesHuUDe704Fr6ssaBA3AzoXpvgOU5veXfzS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711026407, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-21T13:06:47.828 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.803+0000 7faf7bc05700 1 Processor -- start 2024-03-21T13:06:47.828 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.803+0000 7faf7bc05700 1 -- start start 2024-03-21T13:06:47.828 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.803+0000 7faf7bc05700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf74079b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:47.828 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.803+0000 7faf7bc05700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7407a100 con 0x7faf7407b6d0 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.803+0000 7faf799a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf74079b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.803+0000 7faf799a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf74079b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43474/0 (socket says 172.21.15.89:43474) 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.803+0000 7faf799a1700 1 -- 172.21.15.89:0/1420830517 learned_addr learned my addr 172.21.15.89:0/1420830517 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf799a1700 1 -- 172.21.15.89:0/1420830517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7407a280 con 0x7faf7407b6d0 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf799a1700 1 --2- 172.21.15.89:0/1420830517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf74079b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf700098a0 tx=0x7faf70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b44625ca01ac5fa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7899f700 1 -- 172.21.15.89:0/1420830517 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf70004160 con 0x7faf7407b6d0 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7899f700 1 -- 172.21.15.89:0/1420830517 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7faf70004300 con 0x7faf7407b6d0 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7899f700 1 -- 172.21.15.89:0/1420830517 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf700118f0 con 0x7faf7407b6d0 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7bc05700 1 -- 172.21.15.89:0/1420830517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 msgr2=0x7faf74079b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7bc05700 1 --2- 172.21.15.89:0/1420830517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf74079b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf700098a0 tx=0x7faf70009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7bc05700 1 -- 172.21.15.89:0/1420830517 shutdown_connections 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7bc05700 1 --2- 172.21.15.89:0/1420830517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf74079b30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.804+0000 7faf7bc05700 1 -- 172.21.15.89:0/1420830517 >> 172.21.15.89:0/1420830517 conn(0x7faf741014b0 msgr2=0x7faf741038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:47.829 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf7bc05700 1 -- 172.21.15.89:0/1420830517 shutdown_connections 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf7bc05700 1 -- 172.21.15.89:0/1420830517 wait complete. 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf7bc05700 1 Processor -- start 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf7bc05700 1 -- start start 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf7bc05700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf7419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf7bc05700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7410c410 con 0x7faf7407b6d0 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf799a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf7419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf799a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf7419bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43476/0 (socket says 172.21.15.89:43476) 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.805+0000 7faf799a1700 1 -- 172.21.15.89:0/1646078212 learned_addr learned my addr 172.21.15.89:0/1646078212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.806+0000 7faf799a1700 1 -- 172.21.15.89:0/1646078212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7419c480 con 0x7faf7407b6d0 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.806+0000 7faf799a1700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf7419bf40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf70009fb0 tx=0x7faf70011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.806+0000 7faf6affd700 1 -- 172.21.15.89:0/1646078212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf7001a430 con 0x7faf7407b6d0 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.806+0000 7faf6affd700 1 -- 172.21.15.89:0/1646078212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7faf7001aa50 con 0x7faf7407b6d0 2024-03-21T13:06:47.830 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.806+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7419c710 con 0x7faf7407b6d0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.806+0000 7faf6affd700 1 -- 172.21.15.89:0/1646078212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf70019910 con 0x7faf7407b6d0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.806+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7419f400 con 0x7faf7407b6d0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.807+0000 7faf6affd700 1 -- 172.21.15.89:0/1646078212 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 46307+0+0 (secure 0 0 0) 0x7faf70022020 con 0x7faf7407b6d0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.807+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf740637c0 con 0x7faf7407b6d0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.807+0000 7faf6affd700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7faf60039180 0x7faf6003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.807+0000 7faf6affd700 1 -- 172.21.15.89:0/1646078212 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faf7004d8c0 con 0x7faf7407b6d0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.807+0000 7faf791a0700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7faf60039180 0x7faf6003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.807+0000 7faf791a0700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7faf60039180 0x7faf6003b640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faf64006fd0 tx=0x7faf64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:46.812+0000 7faf6affd700 1 -- 172.21.15.89:0/1646078212 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf7002b350 con 0x7faf7407b6d0 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.050+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7faf7419cbf0 con 0x7faf60039180 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.278+0000 7faf6affd700 1 -- 172.21.15.89:0/1646078212 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7faf7419cbf0 con 0x7faf60039180 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7faf60039180 msgr2=0x7faf6003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:47.831 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7faf60039180 0x7faf6003b640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faf64006fd0 tx=0x7faf64006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 msgr2=0x7faf7419bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf7419bf40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf70009fb0 tx=0x7faf70011a20 comp rx=0 tx=0).stop 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 shutdown_connections 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7faf60039180 0x7faf6003b640 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 --2- 172.21.15.89:0/1646078212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf7407b6d0 0x7faf7419bf40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.280+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 >> 172.21.15.89:0/1646078212 conn(0x7faf741014b0 msgr2=0x7faf74101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.281+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 shutdown_connections 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:47.281+0000 7faf7bc05700 1 -- 172.21.15.89:0/1646078212 wait complete. 2024-03-21T13:06:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:Fetching dashboard port number... 2024-03-21T13:06:48.494 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:48 smithi089 ceph-mon[30095]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:48.495 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:49.165 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 8443 2024-03-21T13:06:49.165 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.308+0000 7f6462730700 1 Processor -- start 2024-03-21T13:06:49.165 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.308+0000 7f6462730700 1 -- start start 2024-03-21T13:06:49.165 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.308+0000 7f6462730700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c106280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:49.166 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.308+0000 7f6462730700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c1067c0 con 0x7f645c105e60 2024-03-21T13:06:49.167 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:49.167 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48176/0 (socket says 172.21.15.89:48176) 2024-03-21T13:06:49.167 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645bfff700 1 -- 172.21.15.89:0/678678627 learned_addr learned my addr 172.21.15.89:0/678678627 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:49.167 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645bfff700 1 -- 172.21.15.89:0/678678627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c107050 con 0x7f645c105e60 2024-03-21T13:06:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645bfff700 1 --2- 172.21.15.89:0/678678627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c106280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f644c0098a0 tx=0x7f644c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb3d0b06dc56929 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645affd700 1 -- 172.21.15.89:0/678678627 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f644c004160 con 0x7f645c105e60 2024-03-21T13:06:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645affd700 1 -- 172.21.15.89:0/678678627 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f644c004300 con 0x7f645c105e60 2024-03-21T13:06:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.309+0000 7f645affd700 1 -- 172.21.15.89:0/678678627 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f644c0118f0 con 0x7f645c105e60 2024-03-21T13:06:49.184 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 -- 172.21.15.89:0/678678627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 msgr2=0x7f645c106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 --2- 172.21.15.89:0/678678627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c106280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f644c0098a0 tx=0x7f644c009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 -- 172.21.15.89:0/678678627 shutdown_connections 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 --2- 172.21.15.89:0/678678627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c106280 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 -- 172.21.15.89:0/678678627 >> 172.21.15.89:0/678678627 conn(0x7f645c1014b0 msgr2=0x7f645c1038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 -- 172.21.15.89:0/678678627 shutdown_connections 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 -- 172.21.15.89:0/678678627 wait complete. 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.310+0000 7f6462730700 1 Processor -- start 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f6462730700 1 -- start start 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f6462730700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f6462730700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c106940 con 0x7f645c105e60 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f645bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f645bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48184/0 (socket says 172.21.15.89:48184) 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f645bfff700 1 -- 172.21.15.89:0/1714440541 learned_addr learned my addr 172.21.15.89:0/1714440541 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f645bfff700 1 -- 172.21.15.89:0/1714440541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c19a220 con 0x7f645c105e60 2024-03-21T13:06:49.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f645bfff700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c199ce0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f644c003f60 tx=0x7f644c011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.311+0000 7f64597fa700 1 -- 172.21.15.89:0/1714440541 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f644c0038b0 con 0x7f645c105e60 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.312+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c19a4b0 con 0x7f645c105e60 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.312+0000 7f64597fa700 1 -- 172.21.15.89:0/1714440541 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f644c003a50 con 0x7f645c105e60 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.312+0000 7f64597fa700 1 -- 172.21.15.89:0/1714440541 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f644c024410 con 0x7f645c105e60 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.312+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c19d190 con 0x7f645c105e60 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.312+0000 7f64597fa700 1 -- 172.21.15.89:0/1714440541 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f644c022020 con 0x7f645c105e60 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.312+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c0637c0 con 0x7f645c105e60 2024-03-21T13:06:49.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.313+0000 7f64597fa700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f64440391d0 0x7f644403b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.313+0000 7f64597fa700 1 -- 172.21.15.89:0/1714440541 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f644c04d3b0 con 0x7f645c105e60 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.313+0000 7f645b7fe700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f64440391d0 0x7f644403b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.313+0000 7f645b7fe700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f64440391d0 0x7f644403b690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6450006fd0 tx=0x7f6450006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.318+0000 7f64597fa700 1 -- 172.21.15.89:0/1714440541 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f644c018350 con 0x7f645c105e60 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.534+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f645c19d570 con 0x7f645c105e60 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.535+0000 7f64597fa700 1 -- 172.21.15.89:0/1714440541 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f644c00bd80 con 0x7f645c105e60 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f64440391d0 msgr2=0x7f644403b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f64440391d0 0x7f644403b690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6450006fd0 tx=0x7f6450006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 msgr2=0x7f645c199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c199ce0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f644c003f60 tx=0x7f644c011ef0 comp rx=0 tx=0).stop 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 shutdown_connections 2024-03-21T13:06:49.190 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f64440391d0 0x7f644403b690 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 --2- 172.21.15.89:0/1714440541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f645c105e60 0x7f645c199ce0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 >> 172.21.15.89:0/1714440541 conn(0x7f645c1014b0 msgr2=0x7f645c101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 shutdown_connections 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:48.537+0000 7f6462730700 1 -- 172.21.15.89:0/1714440541 wait complete. 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph Dashboard is now available at: 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:49.191 INFO:teuthology.orchestra.run.smithi089.stdout: URL: https://smithi089.front.sepia.ceph.com:8443/ 2024-03-21T13:06:49.192 INFO:teuthology.orchestra.run.smithi089.stdout: User: admin 2024-03-21T13:06:49.192 INFO:teuthology.orchestra.run.smithi089.stdout: Password: uslui9fozd 2024-03-21T13:06:49.192 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:49.192 INFO:teuthology.orchestra.run.smithi089.stdout:Saving cluster configuration to /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config directory 2024-03-21T13:06:49.192 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling autotune for osd_memory_target 2024-03-21T13:06:49.418 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1714440541' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-21T13:06:50.564 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:50 smithi089 ceph-mon[30095]: mgrmap e13: y(active, since 6s) 2024-03-21T13:06:51.121 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.354+0000 7f5696e70700 1 Processor -- start 2024-03-21T13:06:51.121 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.354+0000 7f5696e70700 1 -- start start 2024-03-21T13:06:51.121 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.354+0000 7f5696e70700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f5690106290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:51.121 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.354+0000 7f5696e70700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56901067d0 con 0x7f5690105e70 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.354+0000 7f5694c0c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f5690106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.354+0000 7f5694c0c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f5690106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48192/0 (socket says 172.21.15.89:48192) 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.354+0000 7f5694c0c700 1 -- 172.21.15.89:0/2315009787 learned_addr learned my addr 172.21.15.89:0/2315009787 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5694c0c700 1 -- 172.21.15.89:0/2315009787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690107060 con 0x7f5690105e70 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5694c0c700 1 --2- 172.21.15.89:0/2315009787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f5690106290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f56840098a0 tx=0x7f5684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a9c5ce6f4dea24 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f568f7fe700 1 -- 172.21.15.89:0/2315009787 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5684004160 con 0x7f5690105e70 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f568f7fe700 1 -- 172.21.15.89:0/2315009787 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5684004300 con 0x7f5690105e70 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f568f7fe700 1 -- 172.21.15.89:0/2315009787 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56840118f0 con 0x7f5690105e70 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5696e70700 1 -- 172.21.15.89:0/2315009787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 msgr2=0x7f5690106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5696e70700 1 --2- 172.21.15.89:0/2315009787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f5690106290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f56840098a0 tx=0x7f5684009bb0 comp rx=0 tx=0).stop 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5696e70700 1 -- 172.21.15.89:0/2315009787 shutdown_connections 2024-03-21T13:06:51.122 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5696e70700 1 --2- 172.21.15.89:0/2315009787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f5690106290 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5696e70700 1 -- 172.21.15.89:0/2315009787 >> 172.21.15.89:0/2315009787 conn(0x7f5690101460 msgr2=0x7f56901038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5696e70700 1 -- 172.21.15.89:0/2315009787 shutdown_connections 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.355+0000 7f5696e70700 1 -- 172.21.15.89:0/2315009787 wait complete. 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5696e70700 1 Processor -- start 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5696e70700 1 -- start start 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5696e70700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f569007b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5696e70700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5690106950 con 0x7f5690105e70 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5694c0c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f569007b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5694c0c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f569007b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48196/0 (socket says 172.21.15.89:48196) 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5694c0c700 1 -- 172.21.15.89:0/3702206572 learned_addr learned my addr 172.21.15.89:0/3702206572 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.356+0000 7f5694c0c700 1 -- 172.21.15.89:0/3702206572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569007b8c0 con 0x7f5690105e70 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.357+0000 7f5694c0c700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f569007b380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5684009fb0 tx=0x7f5684011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.357+0000 7f568dffb700 1 -- 172.21.15.89:0/3702206572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f568401a430 con 0x7f5690105e70 2024-03-21T13:06:51.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.357+0000 7f568dffb700 1 -- 172.21.15.89:0/3702206572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f568401aa50 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.357+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5690079ab0 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.357+0000 7f568dffb700 1 -- 172.21.15.89:0/3702206572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5684019910 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.357+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5690079ff0 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.358+0000 7f568dffb700 1 -- 172.21.15.89:0/3702206572 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f5684022020 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.358+0000 7f568dffb700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f5680039220 0x7f568003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.358+0000 7f568dffb700 1 -- 172.21.15.89:0/3702206572 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f568404d9b0 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.358+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56900637c0 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.358+0000 7f568ffff700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f5680039220 0x7f568003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.358+0000 7f568ffff700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f5680039220 0x7f568003b6e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f567c006fd0 tx=0x7f567c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.363+0000 7f568dffb700 1 -- 172.21.15.89:0/3702206572 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5684029250 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.563+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f569007a6a0 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.564+0000 7f568dffb700 1 -- 172.21.15.89:0/3702206572 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5684020e80 con 0x7f5690105e70 2024-03-21T13:06:51.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f5680039220 msgr2=0x7f568003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f5680039220 0x7f568003b6e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f567c006fd0 tx=0x7f567c006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 msgr2=0x7f569007b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f569007b380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5684009fb0 tx=0x7f5684011a20 comp rx=0 tx=0).stop 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 shutdown_connections 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f5680039220 0x7f568003b6e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 --2- 172.21.15.89:0/3702206572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5690105e70 0x7f569007b380 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 >> 172.21.15.89:0/3702206572 conn(0x7f5690101460 msgr2=0x7f5690101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 shutdown_connections 2024-03-21T13:06:51.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:50.566+0000 7f5696e70700 1 -- 172.21.15.89:0/3702206572 wait complete. 2024-03-21T13:06:52.461 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.568+0000 7fadedc9a700 1 Processor -- start 2024-03-21T13:06:52.461 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.568+0000 7fadedc9a700 1 -- start start 2024-03-21T13:06:52.461 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.568+0000 7fadedc9a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade8106270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.568+0000 7fadedc9a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade81067b0 con 0x7fade8105e50 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.568+0000 7fade77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade8106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.568+0000 7fade77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade8106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48204/0 (socket says 172.21.15.89:48204) 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.568+0000 7fade77fe700 1 -- 172.21.15.89:0/745830465 learned_addr learned my addr 172.21.15.89:0/745830465 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fade77fe700 1 -- 172.21.15.89:0/745830465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade8107040 con 0x7fade8105e50 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fade77fe700 1 --2- 172.21.15.89:0/745830465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade8106270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009e80 tx=0x7fadd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3aa0bc8736a31a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fade67fc700 1 -- 172.21.15.89:0/745830465 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd8017070 con 0x7fade8105e50 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fade67fc700 1 -- 172.21.15.89:0/745830465 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fadd800b8c0 con 0x7fade8105e50 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fade67fc700 1 -- 172.21.15.89:0/745830465 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd8004370 con 0x7fade8105e50 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fadedc9a700 1 -- 172.21.15.89:0/745830465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 msgr2=0x7fade8106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:52.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fadedc9a700 1 --2- 172.21.15.89:0/745830465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade8106270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009e80 tx=0x7fadd800b330 comp rx=0 tx=0).stop 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.569+0000 7fadedc9a700 1 -- 172.21.15.89:0/745830465 shutdown_connections 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 --2- 172.21.15.89:0/745830465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade8106270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 -- 172.21.15.89:0/745830465 >> 172.21.15.89:0/745830465 conn(0x7fade8101460 msgr2=0x7fade8103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 -- 172.21.15.89:0/745830465 shutdown_connections 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 -- 172.21.15.89:0/745830465 wait complete. 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 Processor -- start 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 -- start start 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade807b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.570+0000 7fadedc9a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade8106930 con 0x7fade8105e50 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade807b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade807b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48206/0 (socket says 172.21.15.89:48206) 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade77fe700 1 -- 172.21.15.89:0/3761905262 learned_addr learned my addr 172.21.15.89:0/3761905262 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade77fe700 1 -- 172.21.15.89:0/3761905262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade807b8c0 con 0x7fade8105e50 2024-03-21T13:06:52.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade77fe700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade807b380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fadd800bfa0 tx=0x7fadd8003c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade4ff9700 1 -- 172.21.15.89:0/3761905262 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd8021020 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade4ff9700 1 -- 172.21.15.89:0/3761905262 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fadd800ba70 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade8079ab0 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fade4ff9700 1 -- 172.21.15.89:0/3761905262 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd8017040 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.571+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade8079ff0 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.572+0000 7fade4ff9700 1 -- 172.21.15.89:0/3761905262 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7fadd8024020 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.572+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade80637c0 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.572+0000 7fade4ff9700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fadd40392d0 0x7fadd403b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.572+0000 7fade4ff9700 1 -- 172.21.15.89:0/3761905262 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fadd800f050 con 0x7fade8105e50 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.572+0000 7fade6ffd700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fadd40392d0 0x7fadd403b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.573+0000 7fade6ffd700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fadd40392d0 0x7fadd403b790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006fd0 tx=0x7fadd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:52.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.577+0000 7fade4ff9700 1 -- 172.21.15.89:0/3761905262 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fadd802b020 con 0x7fade8105e50 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.880+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fade807a580 con 0x7fade8105e50 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.882+0000 7fade4ff9700 1 -- 172.21.15.89:0/3761905262 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7fadd802c260 con 0x7fade8105e50 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fadd40392d0 msgr2=0x7fadd403b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fadd40392d0 0x7fadd403b790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006fd0 tx=0x7fadd0006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 msgr2=0x7fade807b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade807b380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fadd800bfa0 tx=0x7fadd8003c50 comp rx=0 tx=0).stop 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 shutdown_connections 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fadd40392d0 0x7fadd403b790 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 --2- 172.21.15.89:0/3761905262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade8105e50 0x7fade807b380 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.884+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 >> 172.21.15.89:0/3761905262 conn(0x7fade8101460 msgr2=0x7fade8101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.885+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 shutdown_connections 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-03-21T13:06:51.885+0000 7fadedc9a700 1 -- 172.21.15.89:0/3761905262 wait complete. 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-21T13:06:52.465 INFO:teuthology.orchestra.run.smithi089.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout:Or, if you are only running a single cluster on this host: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: ceph telemetry on 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout:For more information see: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-21T13:06:52.467 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:06:52.467 INFO:teuthology.orchestra.run.smithi089.stdout:Bootstrap complete. 2024-03-21T13:06:52.498 INFO:tasks.cephadm:Fetching config... 2024-03-21T13:06:52.498 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:06:52.499 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-21T13:06:52.515 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-21T13:06:52.515 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:06:52.515 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-21T13:06:52.571 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-21T13:06:52.571 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:06:52.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-21T13:06:52.638 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-21T13:06:52.638 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:06:52.638 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-21T13:06:52.693 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-21T13:06:52.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8TNS8VfTI9BYnZ8hnyEpHI3TLDktC+SiSLTLNMl0eyJxqVjs6Fsla8tWmkCQ8F/ITbDSeU8x0WwUBRvgfu9Kjm9yOJV41eO+RyRM4ugwas9Sr+uwodvw/5wq+tQzaY4daviCjB38NCKrtsngWXiLpCenngHNWdjzVkp+1BKgrVBoTtAccj036hjt2O2kSZbQxkIHeja5GYdoOfod2hBppzYFMvCQK/GOdI6zUV0/UR0RetW3r1qF0zKZ9HMwMxMXRqpgpetoxI906lFjhumkaOj7bL3ybdcm5pxoXS9RvFIhpBKxO0+FmKBaaZXBuG7v1b/lBCZzR5yObtZwolZ8LP+Y4+FkHBfEEFD8TsHkSlkCFO6JKNXIcWSrRvRuB5x3nlMwm4hpC3QW3nrLwol5p75yEZSaanGlgHJpfWMRkLRjql9LAsuMbRO1k+nAsWDN6J/XxECEVN76TO+oBPxeOib//vJtLVSDSeWnEFSfBSWexHS6OFumrx1GOSu+Ow8= ceph-9c355834-e783-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T13:06:52.780 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8TNS8VfTI9BYnZ8hnyEpHI3TLDktC+SiSLTLNMl0eyJxqVjs6Fsla8tWmkCQ8F/ITbDSeU8x0WwUBRvgfu9Kjm9yOJV41eO+RyRM4ugwas9Sr+uwodvw/5wq+tQzaY4daviCjB38NCKrtsngWXiLpCenngHNWdjzVkp+1BKgrVBoTtAccj036hjt2O2kSZbQxkIHeja5GYdoOfod2hBppzYFMvCQK/GOdI6zUV0/UR0RetW3r1qF0zKZ9HMwMxMXRqpgpetoxI906lFjhumkaOj7bL3ybdcm5pxoXS9RvFIhpBKxO0+FmKBaaZXBuG7v1b/lBCZzR5yObtZwolZ8LP+Y4+FkHBfEEFD8TsHkSlkCFO6JKNXIcWSrRvRuB5x3nlMwm4hpC3QW3nrLwol5p75yEZSaanGlgHJpfWMRkLRjql9LAsuMbRO1k+nAsWDN6J/XxECEVN76TO+oBPxeOib//vJtLVSDSeWnEFSfBSWexHS6OFumrx1GOSu+Ow8= ceph-9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:06:52.794 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8TNS8VfTI9BYnZ8hnyEpHI3TLDktC+SiSLTLNMl0eyJxqVjs6Fsla8tWmkCQ8F/ITbDSeU8x0WwUBRvgfu9Kjm9yOJV41eO+RyRM4ugwas9Sr+uwodvw/5wq+tQzaY4daviCjB38NCKrtsngWXiLpCenngHNWdjzVkp+1BKgrVBoTtAccj036hjt2O2kSZbQxkIHeja5GYdoOfod2hBppzYFMvCQK/GOdI6zUV0/UR0RetW3r1qF0zKZ9HMwMxMXRqpgpetoxI906lFjhumkaOj7bL3ybdcm5pxoXS9RvFIhpBKxO0+FmKBaaZXBuG7v1b/lBCZzR5yObtZwolZ8LP+Y4+FkHBfEEFD8TsHkSlkCFO6JKNXIcWSrRvRuB5x3nlMwm4hpC3QW3nrLwol5p75yEZSaanGlgHJpfWMRkLRjql9LAsuMbRO1k+nAsWDN6J/XxECEVN76TO+oBPxeOib//vJtLVSDSeWnEFSfBSWexHS6OFumrx1GOSu+Ow8= ceph-9c355834-e783-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T13:06:52.840 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8TNS8VfTI9BYnZ8hnyEpHI3TLDktC+SiSLTLNMl0eyJxqVjs6Fsla8tWmkCQ8F/ITbDSeU8x0WwUBRvgfu9Kjm9yOJV41eO+RyRM4ugwas9Sr+uwodvw/5wq+tQzaY4daviCjB38NCKrtsngWXiLpCenngHNWdjzVkp+1BKgrVBoTtAccj036hjt2O2kSZbQxkIHeja5GYdoOfod2hBppzYFMvCQK/GOdI6zUV0/UR0RetW3r1qF0zKZ9HMwMxMXRqpgpetoxI906lFjhumkaOj7bL3ybdcm5pxoXS9RvFIhpBKxO0+FmKBaaZXBuG7v1b/lBCZzR5yObtZwolZ8LP+Y4+FkHBfEEFD8TsHkSlkCFO6JKNXIcWSrRvRuB5x3nlMwm4hpC3QW3nrLwol5p75yEZSaanGlgHJpfWMRkLRjql9LAsuMbRO1k+nAsWDN6J/XxECEVN76TO+oBPxeOib//vJtLVSDSeWnEFSfBSWexHS6OFumrx1GOSu+Ow8= ceph-9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:06:52.854 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-21T13:06:53.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3761905262' entity='client.admin' 2024-03-21T13:06:53.093 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.182+0000 7f4dcb006700 1 -- 172.21.15.89:0/2914554733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 msgr2=0x7f4dc4100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.182+0000 7f4dcb006700 1 --2- 172.21.15.89:0/2914554733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc4100f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009910 tx=0x7f4db8009c20 comp rx=0 tx=0).stop 2024-03-21T13:06:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.182+0000 7f4dcb006700 1 -- 172.21.15.89:0/2914554733 shutdown_connections 2024-03-21T13:06:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.182+0000 7f4dcb006700 1 --2- 172.21.15.89:0/2914554733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc4100f20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.182+0000 7f4dcb006700 1 -- 172.21.15.89:0/2914554733 >> 172.21.15.89:0/2914554733 conn(0x7f4dc40fc060 msgr2=0x7f4dc40fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.182+0000 7f4dcb006700 1 -- 172.21.15.89:0/2914554733 shutdown_connections 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.182+0000 7f4dcb006700 1 -- 172.21.15.89:0/2914554733 wait complete. 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dcb006700 1 Processor -- start 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dcb006700 1 -- start start 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dcb006700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc406aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dcb006700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4106000 con 0x7f4dc4100b00 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc8da2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc406aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc8da2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc406aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48226/0 (socket says 172.21.15.89:48226) 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc8da2700 1 -- 172.21.15.89:0/3756902839 learned_addr learned my addr 172.21.15.89:0/3756902839 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc8da2700 1 -- 172.21.15.89:0/3756902839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc4199580 con 0x7f4dc4100b00 2024-03-21T13:06:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc8da2700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc406aa50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4db8005740 tx=0x7f4db8004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc1ffb700 1 -- 172.21.15.89:0/3756902839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4db8027070 con 0x7f4dc4100b00 2024-03-21T13:06:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc1ffb700 1 -- 172.21.15.89:0/3756902839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4db800bde0 con 0x7f4dc4100b00 2024-03-21T13:06:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc4199750 con 0x7f4dc4100b00 2024-03-21T13:06:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.183+0000 7f4dc1ffb700 1 -- 172.21.15.89:0/3756902839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4db800fe30 con 0x7f4dc4100b00 2024-03-21T13:06:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.184+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc4199af0 con 0x7f4dc4100b00 2024-03-21T13:06:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.184+0000 7f4dc1ffb700 1 -- 172.21.15.89:0/3756902839 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f4db802a070 con 0x7f4dc4100b00 2024-03-21T13:06:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.184+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc4106780 con 0x7f4dc4100b00 2024-03-21T13:06:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.185+0000 7f4dc1ffb700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f4db4039280 0x7f4db403b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.185+0000 7f4dc1ffb700 1 -- 172.21.15.89:0/3756902839 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4db804d540 con 0x7f4dc4100b00 2024-03-21T13:06:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.185+0000 7f4dc3fff700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f4db4039280 0x7f4db403b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.185+0000 7f4dc3fff700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f4db4039280 0x7f4db403b740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4db0006fd0 tx=0x7f4db0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:55.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.189+0000 7f4dc1ffb700 1 -- 172.21.15.89:0/3756902839 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4db8013070 con 0x7f4dc4100b00 2024-03-21T13:06:55.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.386+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4dc402da30 con 0x7f4dc4100b00 2024-03-21T13:06:55.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.388+0000 7f4dc1ffb700 1 -- 172.21.15.89:0/3756902839 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4db8013250 con 0x7f4dc4100b00 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f4db4039280 msgr2=0x7f4db403b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f4db4039280 0x7f4db403b740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4db0006fd0 tx=0x7f4db0006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 msgr2=0x7f4dc406aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc406aa50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4db8005740 tx=0x7f4db8004f60 comp rx=0 tx=0).stop 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 shutdown_connections 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f4db4039280 0x7f4db403b740 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 --2- 172.21.15.89:0/3756902839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dc4100b00 0x7f4dc406aa50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 >> 172.21.15.89:0/3756902839 conn(0x7f4dc40fc060 msgr2=0x7f4dc40fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 shutdown_connections 2024-03-21T13:06:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:55.390+0000 7f4dcb006700 1 -- 172.21.15.89:0/3756902839 wait complete. 2024-03-21T13:06:55.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:55 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:55.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:55 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:55.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:55 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:06:55.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:55 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:55.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:55 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:55.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:55 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:56.017 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-21T13:06:56.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-21T13:06:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:56.709 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3756902839' entity='client.admin' 2024-03-21T13:06:56.710 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:56 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:56.710 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:56 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:57.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 -- 172.21.15.89:0/1786120018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 msgr2=0x7fc5fc102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:57.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 --2- 172.21.15.89:0/1786120018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc102000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009910 tx=0x7fc5ec009c20 comp rx=0 tx=0).stop 2024-03-21T13:06:57.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 -- 172.21.15.89:0/1786120018 shutdown_connections 2024-03-21T13:06:57.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 --2- 172.21.15.89:0/1786120018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc102000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:57.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 -- 172.21.15.89:0/1786120018 >> 172.21.15.89:0/1786120018 conn(0x7fc5fc0fd180 msgr2=0x7fc5fc0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 -- 172.21.15.89:0/1786120018 shutdown_connections 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 -- 172.21.15.89:0/1786120018 wait complete. 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 Processor -- start 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.796+0000 7fc602284700 1 -- start start 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc602284700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc075e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc602284700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc1070e0 con 0x7fc5fc101be0 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5fbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc075e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5fbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc075e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42272/0 (socket says 172.21.15.89:42272) 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5fbfff700 1 -- 172.21.15.89:0/887066619 learned_addr learned my addr 172.21.15.89:0/887066619 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5fbfff700 1 -- 172.21.15.89:0/887066619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc076390 con 0x7fc5fc101be0 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5fbfff700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc075e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec005740 tx=0x7fc5ec004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5f9ffb700 1 -- 172.21.15.89:0/887066619 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5ec027070 con 0x7fc5fc101be0 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5f9ffb700 1 -- 172.21.15.89:0/887066619 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc5ec00bde0 con 0x7fc5fc101be0 2024-03-21T13:06:57.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc074580 con 0x7fc5fc101be0 2024-03-21T13:06:57.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc5f9ffb700 1 -- 172.21.15.89:0/887066619 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5ec00fe30 con 0x7fc5fc101be0 2024-03-21T13:06:57.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.797+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc074ac0 con 0x7fc5fc101be0 2024-03-21T13:06:57.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.798+0000 7fc5f9ffb700 1 -- 172.21.15.89:0/887066619 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7fc5ec02a070 con 0x7fc5fc101be0 2024-03-21T13:06:57.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.798+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc107860 con 0x7fc5fc101be0 2024-03-21T13:06:57.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.799+0000 7fc5f9ffb700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc5e8039220 0x7fc5e803b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:06:57.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.799+0000 7fc5f9ffb700 1 -- 172.21.15.89:0/887066619 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc5ec04d450 con 0x7fc5fc101be0 2024-03-21T13:06:57.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.799+0000 7fc5f3fff700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc5e8039220 0x7fc5e803b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:06:57.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.799+0000 7fc5f3fff700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc5e8039220 0x7fc5e803b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4006fd0 tx=0x7fc5e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:06:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:57.803+0000 7fc5f9ffb700 1 -- 172.21.15.89:0/887066619 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc5ec013070 con 0x7fc5fc101be0 2024-03-21T13:06:58.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.000+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5fc18ff30 con 0x7fc5e8039220 2024-03-21T13:06:58.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.003+0000 7fc5f9ffb700 1 -- 172.21.15.89:0/887066619 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc5fc18ff30 con 0x7fc5e8039220 2024-03-21T13:06:58.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc5e8039220 msgr2=0x7fc5e803b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:58.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc5e8039220 0x7fc5e803b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4006fd0 tx=0x7fc5e4006e40 comp rx=0 tx=0).stop 2024-03-21T13:06:58.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 msgr2=0x7fc5fc075e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:06:58.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc075e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec005740 tx=0x7fc5ec004f60 comp rx=0 tx=0).stop 2024-03-21T13:06:58.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 shutdown_connections 2024-03-21T13:06:58.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc5e8039220 0x7fc5e803b6e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:58.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 --2- 172.21.15.89:0/887066619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5fc101be0 0x7fc5fc075e50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:06:58.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 >> 172.21.15.89:0/887066619 conn(0x7fc5fc0fd180 msgr2=0x7fc5fc0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:06:58.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 shutdown_connections 2024-03-21T13:06:58.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:06:58.005+0000 7fc602284700 1 -- 172.21.15.89:0/887066619 wait complete. 2024-03-21T13:06:58.775 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-03-21T13:06:58.775 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:06:58.775 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-03-21T13:06:58.793 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:06:58.793 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:06:58.850 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-03-21T13:06:58.850 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch host add smithi152 2024-03-21T13:06:59.087 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:06:59.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:59.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:59.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:59.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:59.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:59.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:59.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:06:59.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:06:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:00.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:00 smithi089 ceph-mon[30095]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-21T13:07:00.295 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:00 smithi089 ceph-mon[30095]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:00.295 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:00 smithi089 ceph-mon[30095]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:07:00.295 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:00 smithi089 ceph-mon[30095]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:07:00.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.501+0000 7f3a3d58b700 1 -- 172.21.15.89:0/301925921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 msgr2=0x7f3a38103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.501+0000 7f3a3d58b700 1 --2- 172.21.15.89:0/301925921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a38103140 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009910 tx=0x7f3a28009c20 comp rx=0 tx=0).stop 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.501+0000 7f3a3d58b700 1 -- 172.21.15.89:0/301925921 shutdown_connections 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.501+0000 7f3a3d58b700 1 --2- 172.21.15.89:0/301925921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a38103140 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.501+0000 7f3a3d58b700 1 -- 172.21.15.89:0/301925921 >> 172.21.15.89:0/301925921 conn(0x7f3a380fc5d0 msgr2=0x7f3a380fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.501+0000 7f3a3d58b700 1 -- 172.21.15.89:0/301925921 shutdown_connections 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.502+0000 7f3a3d58b700 1 -- 172.21.15.89:0/301925921 wait complete. 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.502+0000 7f3a3d58b700 1 Processor -- start 2024-03-21T13:07:00.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.502+0000 7f3a3d58b700 1 -- start start 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.502+0000 7f3a3d58b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a381009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.502+0000 7f3a3d58b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3810b6a0 con 0x7f3a38102d60 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a36ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a381009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a36ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a381009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42302/0 (socket says 172.21.15.89:42302) 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a36ffd700 1 -- 172.21.15.89:0/2754373741 learned_addr learned my addr 172.21.15.89:0/2754373741 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a36ffd700 1 -- 172.21.15.89:0/2754373741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a381028b0 con 0x7f3a38102d60 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a36ffd700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a381009c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800f690 tx=0x7f3a2800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a34ff9700 1 -- 172.21.15.89:0/2754373741 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a28027070 con 0x7f3a38102d60 2024-03-21T13:07:00.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a34ff9700 1 -- 172.21.15.89:0/2754373741 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3a2800fb90 con 0x7f3a38102d60 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38100f00 con 0x7f3a38102d60 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a34ff9700 1 -- 172.21.15.89:0/2754373741 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a28017440 con 0x7f3a38102d60 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.503+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a381013c0 con 0x7f3a38102d60 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.504+0000 7f3a34ff9700 1 -- 172.21.15.89:0/2754373741 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f3a2802a070 con 0x7f3a38102d60 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.504+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3806a8b0 con 0x7f3a38102d60 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.504+0000 7f3a34ff9700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f3a24039220 0x7f3a2403b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.504+0000 7f3a34ff9700 1 -- 172.21.15.89:0/2754373741 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3a2801e030 con 0x7f3a38102d60 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.504+0000 7f3a2e7fc700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f3a24039220 0x7f3a2403b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:00.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.505+0000 7f3a2e7fc700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f3a24039220 0x7f3a2403b6e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a20006fd0 tx=0x7f3a20006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:00.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.509+0000 7f3a34ff9700 1 -- 172.21.15.89:0/2754373741 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a28013070 con 0x7f3a38102d60 2024-03-21T13:07:00.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:00.709+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a380637c0 con 0x7f3a24039220 2024-03-21T13:07:01.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:01 smithi089 ceph-mon[30095]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:02.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:02 smithi089 ceph-mon[30095]: Deploying cephadm binary to smithi152 2024-03-21T13:07:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:03 smithi089 ceph-mon[30095]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:04.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.910+0000 7f3a34ff9700 1 -- 172.21.15.89:0/2754373741 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3a380637c0 con 0x7f3a24039220 2024-03-21T13:07:04.911 INFO:teuthology.orchestra.run.smithi089.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f3a24039220 msgr2=0x7f3a2403b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f3a24039220 0x7f3a2403b6e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a20006fd0 tx=0x7f3a20006e40 comp rx=0 tx=0).stop 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 msgr2=0x7f3a381009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a381009c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800f690 tx=0x7f3a2800f770 comp rx=0 tx=0).stop 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 shutdown_connections 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f3a24039220 0x7f3a2403b6e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 --2- 172.21.15.89:0/2754373741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a38102d60 0x7f3a381009c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.912+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 >> 172.21.15.89:0/2754373741 conn(0x7f3a380fc5d0 msgr2=0x7f3a381063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.913+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 shutdown_connections 2024-03-21T13:07:04.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:04.913+0000 7f3a3d58b700 1 -- 172.21.15.89:0/2754373741 wait complete. 2024-03-21T13:07:05.581 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-21T13:07:05.819 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:06.212 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:06.212 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:05 smithi089 ceph-mon[30095]: Added host smithi152 2024-03-21T13:07:06.212 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:05 smithi089 ceph-mon[30095]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:06.212 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:06.212 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:07.193 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:06 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:07.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.418+0000 7f6052e06700 1 -- 172.21.15.89:0/2472045840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 msgr2=0x7f604c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:07.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.418+0000 7f6052e06700 1 --2- 172.21.15.89:0/2472045840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c106e50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6040009910 tx=0x7f6040009c20 comp rx=0 tx=0).stop 2024-03-21T13:07:07.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.418+0000 7f6052e06700 1 -- 172.21.15.89:0/2472045840 shutdown_connections 2024-03-21T13:07:07.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.418+0000 7f6052e06700 1 --2- 172.21.15.89:0/2472045840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c106e50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:07.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.418+0000 7f6052e06700 1 -- 172.21.15.89:0/2472045840 >> 172.21.15.89:0/2472045840 conn(0x7f604c0fc580 msgr2=0x7f604c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:07.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.418+0000 7f6052e06700 1 -- 172.21.15.89:0/2472045840 shutdown_connections 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.418+0000 7f6052e06700 1 -- 172.21.15.89:0/2472045840 wait complete. 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.419+0000 7f6052e06700 1 Processor -- start 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.419+0000 7f6052e06700 1 -- start start 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.419+0000 7f6052e06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c19b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.419+0000 7f6052e06700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c10a530 con 0x7f604c106a70 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.419+0000 7f6050ba2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c19b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.419+0000 7f6050ba2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c19b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42332/0 (socket says 172.21.15.89:42332) 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.419+0000 7f6050ba2700 1 -- 172.21.15.89:0/1890772656 learned_addr learned my addr 172.21.15.89:0/1890772656 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.420+0000 7f6050ba2700 1 -- 172.21.15.89:0/1890772656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c19bc90 con 0x7f604c106a70 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.420+0000 7f6050ba2700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c19b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6040006010 tx=0x7f604000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:07.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.420+0000 7f6049ffb700 1 -- 172.21.15.89:0/1890772656 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6040027070 con 0x7f604c106a70 2024-03-21T13:07:07.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.420+0000 7f6049ffb700 1 -- 172.21.15.89:0/1890772656 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6040022470 con 0x7f604c106a70 2024-03-21T13:07:07.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.420+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c19bf80 con 0x7f604c106a70 2024-03-21T13:07:07.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.420+0000 7f6049ffb700 1 -- 172.21.15.89:0/1890772656 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6040017440 con 0x7f604c106a70 2024-03-21T13:07:07.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.420+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c195910 con 0x7f604c106a70 2024-03-21T13:07:07.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.421+0000 7f6049ffb700 1 -- 172.21.15.89:0/1890772656 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f604002a070 con 0x7f604c106a70 2024-03-21T13:07:07.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.421+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c1021b0 con 0x7f604c106a70 2024-03-21T13:07:07.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.421+0000 7f6049ffb700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f603c039280 0x7f603c03b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:07.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.421+0000 7f6049ffb700 1 -- 172.21.15.89:0/1890772656 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f604001e030 con 0x7f604c106a70 2024-03-21T13:07:07.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.421+0000 7f604bfff700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f603c039280 0x7f603c03b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:07.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.422+0000 7f604bfff700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f603c039280 0x7f603c03b740 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6038006fd0 tx=0x7f6038006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:07.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.426+0000 7f6049ffb700 1 -- 172.21.15.89:0/1890772656 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6040013070 con 0x7f604c106a70 2024-03-21T13:07:07.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.628+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f604c196e40 con 0x7f603c039280 2024-03-21T13:07:07.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.629+0000 7f6049ffb700 1 -- 172.21.15.89:0/1890772656 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f604c196e40 con 0x7f603c039280 2024-03-21T13:07:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:07:07.629 INFO:teuthology.orchestra.run.smithi089.stdout:[{"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f603c039280 msgr2=0x7f603c03b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f603c039280 0x7f603c03b740 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6038006fd0 tx=0x7f6038006e40 comp rx=0 tx=0).stop 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 msgr2=0x7f604c19b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c19b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6040006010 tx=0x7f604000f740 comp rx=0 tx=0).stop 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 shutdown_connections 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f603c039280 0x7f603c03b740 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 --2- 172.21.15.89:0/1890772656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f604c106a70 0x7f604c19b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 >> 172.21.15.89:0/1890772656 conn(0x7f604c0fc580 msgr2=0x7f604c108a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 shutdown_connections 2024-03-21T13:07:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:07.631+0000 7f6052e06700 1 -- 172.21.15.89:0/1890772656 wait complete. 2024-03-21T13:07:08.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:07 smithi089 ceph-mon[30095]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:08.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:08.225 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-21T13:07:08.225 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-21T13:07:08.462 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:09.319 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:08 smithi089 ceph-mon[30095]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 -- 172.21.15.89:0/3415472147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 msgr2=0x7feb70106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 --2- 172.21.15.89:0/3415472147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb70106e40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb60009910 tx=0x7feb60009c20 comp rx=0 tx=0).stop 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 -- 172.21.15.89:0/3415472147 shutdown_connections 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 --2- 172.21.15.89:0/3415472147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb70106e40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 -- 172.21.15.89:0/3415472147 >> 172.21.15.89:0/3415472147 conn(0x7feb700fc530 msgr2=0x7feb700fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 -- 172.21.15.89:0/3415472147 shutdown_connections 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 -- 172.21.15.89:0/3415472147 wait complete. 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.067+0000 7feb75c18700 1 Processor -- start 2024-03-21T13:07:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb75c18700 1 -- start start 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb75c18700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb701979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb75c18700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7010a520 con 0x7feb70106a60 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb701979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb701979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52696/0 (socket says 172.21.15.89:52696) 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6f7fe700 1 -- 172.21.15.89:0/762272355 learned_addr learned my addr 172.21.15.89:0/762272355 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6f7fe700 1 -- 172.21.15.89:0/762272355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb70197f30 con 0x7feb70106a60 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6f7fe700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb701979f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb60006010 tx=0x7feb6000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6cff9700 1 -- 172.21.15.89:0/762272355 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb60027070 con 0x7feb70106a60 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6cff9700 1 -- 172.21.15.89:0/762272355 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7feb60022470 con 0x7feb70106a60 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb70198220 con 0x7feb70106a60 2024-03-21T13:07:10.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb6cff9700 1 -- 172.21.15.89:0/762272355 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb60017440 con 0x7feb70106a60 2024-03-21T13:07:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.068+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb701986e0 con 0x7feb70106a60 2024-03-21T13:07:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.069+0000 7feb6cff9700 1 -- 172.21.15.89:0/762272355 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7feb6002a070 con 0x7feb70106a60 2024-03-21T13:07:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.069+0000 7feb6cff9700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7feb58039220 0x7feb5803b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.069+0000 7feb6cff9700 1 -- 172.21.15.89:0/762272355 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feb6001e030 con 0x7feb70106a60 2024-03-21T13:07:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.069+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb701021a0 con 0x7feb70106a60 2024-03-21T13:07:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.070+0000 7feb6effd700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7feb58039220 0x7feb5803b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.070+0000 7feb6effd700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7feb58039220 0x7feb5803b6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb64006fd0 tx=0x7feb64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:10.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.075+0000 7feb6cff9700 1 -- 172.21.15.89:0/762272355 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb60013070 con 0x7feb70106a60 2024-03-21T13:07:10.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.274+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7feb700637c0 con 0x7feb70106a60 2024-03-21T13:07:10.275 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:09 smithi089 ceph-mon[30095]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.944+0000 7feb6cff9700 1 -- 172.21.15.89:0/762272355 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7feb6000cdc0 con 0x7feb70106a60 2024-03-21T13:07:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7feb58039220 msgr2=0x7feb5803b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7feb58039220 0x7feb5803b6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb64006fd0 tx=0x7feb64006e40 comp rx=0 tx=0).stop 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 msgr2=0x7feb701979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb701979f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb60006010 tx=0x7feb6000f740 comp rx=0 tx=0).stop 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 shutdown_connections 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7feb58039220 0x7feb5803b6e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 --2- 172.21.15.89:0/762272355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70106a60 0x7feb701979f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 >> 172.21.15.89:0/762272355 conn(0x7feb700fc530 msgr2=0x7feb701089a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 shutdown_connections 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:10.947+0000 7feb75c18700 1 -- 172.21.15.89:0/762272355 wait complete. 2024-03-21T13:07:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-03-21T13:07:11.317 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/762272355' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T13:07:11.317 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:11.317 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:11.317 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:11.318 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:11.318 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:07:11.318 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:11.318 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:10 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:11.652 INFO:tasks.cephadm:Adding mon.a on smithi089 2024-03-21T13:07:11.652 INFO:tasks.cephadm:Adding mon.c on smithi089 2024-03-21T13:07:11.653 INFO:tasks.cephadm:Adding mon.b on smithi152 2024-03-21T13:07:11.653 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b' 2024-03-21T13:07:11.886 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:11 smithi089 ceph-mon[30095]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:07:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:11 smithi089 ceph-mon[30095]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/762272355' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T13:07:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:11 smithi089 ceph-mon[30095]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:11 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:11 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:11 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:13.196 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.195+0000 7f956d4db700 1 -- 172.21.15.152:0/933062523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 msgr2=0x7f9568106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:13.196 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.195+0000 7f956d4db700 1 --2- 172.21.15.152:0/933062523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f9568106e40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f955c009910 tx=0x7f955c009c20 comp rx=0 tx=0).stop 2024-03-21T13:07:13.196 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.195+0000 7f956d4db700 1 -- 172.21.15.152:0/933062523 shutdown_connections 2024-03-21T13:07:13.196 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.195+0000 7f956d4db700 1 --2- 172.21.15.152:0/933062523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f9568106e40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:13.196 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.195+0000 7f956d4db700 1 -- 172.21.15.152:0/933062523 >> 172.21.15.152:0/933062523 conn(0x7f95680fc550 msgr2=0x7f95680fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:13.197 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.195+0000 7f956d4db700 1 -- 172.21.15.152:0/933062523 shutdown_connections 2024-03-21T13:07:13.197 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.195+0000 7f956d4db700 1 -- 172.21.15.152:0/933062523 wait complete. 2024-03-21T13:07:13.197 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.196+0000 7f956d4db700 1 Processor -- start 2024-03-21T13:07:13.197 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.196+0000 7f956d4db700 1 -- start start 2024-03-21T13:07:13.198 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.196+0000 7f956d4db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f956819b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:13.198 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.196+0000 7f956d4db700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956810a520 con 0x7f9568106a60 2024-03-21T13:07:13.198 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.196+0000 7f9566ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f956819b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:13.198 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.196+0000 7f9566ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f956819b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.152:43432/0 (socket says 172.21.15.152:43432) 2024-03-21T13:07:13.198 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f9566ffd700 1 -- 172.21.15.152:0/3975204826 learned_addr learned my addr 172.21.15.152:0/3975204826 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:07:13.198 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f9566ffd700 1 -- 172.21.15.152:0/3975204826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956819bc60 con 0x7f9568106a60 2024-03-21T13:07:13.199 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f9566ffd700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f956819b720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f955c006010 tx=0x7f955c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:13.199 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f954ffff700 1 -- 172.21.15.152:0/3975204826 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f955c027070 con 0x7f9568106a60 2024-03-21T13:07:13.199 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956819bf50 con 0x7f9568106a60 2024-03-21T13:07:13.199 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f954ffff700 1 -- 172.21.15.152:0/3975204826 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f955c022470 con 0x7f9568106a60 2024-03-21T13:07:13.199 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f954ffff700 1 -- 172.21.15.152:0/3975204826 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f955c017440 con 0x7f9568106a60 2024-03-21T13:07:13.199 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.197+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95681958e0 con 0x7f9568106a60 2024-03-21T13:07:13.200 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.198+0000 7f954ffff700 1 -- 172.21.15.152:0/3975204826 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f955c02a070 con 0x7f9568106a60 2024-03-21T13:07:13.200 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.198+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95681021a0 con 0x7f9568106a60 2024-03-21T13:07:13.200 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.198+0000 7f954ffff700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f95500392d0 0x7f955003b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:13.200 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.198+0000 7f954ffff700 1 -- 172.21.15.152:0/3975204826 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f955c01e030 con 0x7f9568106a60 2024-03-21T13:07:13.200 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.199+0000 7f95667fc700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f95500392d0 0x7f955003b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:13.200 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.199+0000 7f95667fc700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f95500392d0 0x7f955003b790 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9558006fd0 tx=0x7f9558006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:13.204 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.204+0000 7f954ffff700 1 -- 172.21.15.152:0/3975204826 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f955c013070 con 0x7f9568106a60 2024-03-21T13:07:13.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:12 smithi089 ceph-mon[30095]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:12 smithi089 ceph-mon[30095]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:07:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:12 smithi089 ceph-mon[30095]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:07:13.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.411+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f9568196e10 con 0x7f95500392d0 2024-03-21T13:07:13.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.420+0000 7f954ffff700 1 -- 172.21.15.152:0/3975204826 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9568196e10 con 0x7f95500392d0 2024-03-21T13:07:13.421 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mon update... 2024-03-21T13:07:13.423 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f95500392d0 msgr2=0x7f955003b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:13.423 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f95500392d0 0x7f955003b790 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9558006fd0 tx=0x7f9558006e40 comp rx=0 tx=0).stop 2024-03-21T13:07:13.423 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 msgr2=0x7f956819b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:13.423 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f956819b720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f955c006010 tx=0x7f955c00f740 comp rx=0 tx=0).stop 2024-03-21T13:07:13.423 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 shutdown_connections 2024-03-21T13:07:13.424 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f95500392d0 0x7f955003b790 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:13.424 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 --2- 172.21.15.152:0/3975204826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9568106a60 0x7f956819b720 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:13.424 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 >> 172.21.15.152:0/3975204826 conn(0x7f95680fc550 msgr2=0x7f9568108920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:13.424 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 shutdown_connections 2024-03-21T13:07:13.424 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:13.422+0000 7f956d4db700 1 -- 172.21.15.152:0/3975204826 wait complete. 2024-03-21T13:07:13.994 DEBUG:teuthology.orchestra.run.smithi089:mon.c> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.c.service 2024-03-21T13:07:13.997 DEBUG:teuthology.orchestra.run.smithi152:mon.b> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.b.service 2024-03-21T13:07:13.999 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T13:07:14.000 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T13:07:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:14.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:14.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:14.742 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:14 smithi089 ceph-mon[30095]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:14 smithi089 ceph-mon[30095]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b;count:3 2024-03-21T13:07:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:14 smithi089 ceph-mon[30095]: Deploying daemon mon.b on smithi152 2024-03-21T13:07:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:15 smithi089 ceph-mon[30095]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.355+0000 7f8c2492e700 1 -- 172.21.15.152:0/3419780951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 msgr2=0x7f8c20106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.355+0000 7f8c2492e700 1 --2- 172.21.15.152:0/3419780951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20106e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009910 tx=0x7f8c10009c20 comp rx=0 tx=0).stop 2024-03-21T13:07:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.355+0000 7f8c2492e700 1 -- 172.21.15.152:0/3419780951 shutdown_connections 2024-03-21T13:07:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.355+0000 7f8c2492e700 1 --2- 172.21.15.152:0/3419780951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20106e50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.355+0000 7f8c2492e700 1 -- 172.21.15.152:0/3419780951 >> 172.21.15.152:0/3419780951 conn(0x7f8c200fc580 msgr2=0x7f8c200fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.355+0000 7f8c2492e700 1 -- 172.21.15.152:0/3419780951 shutdown_connections 2024-03-21T13:07:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.355+0000 7f8c2492e700 1 -- 172.21.15.152:0/3419780951 wait complete. 2024-03-21T13:07:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.356+0000 7f8c2492e700 1 Processor -- start 2024-03-21T13:07:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.356+0000 7f8c2492e700 1 -- start start 2024-03-21T13:07:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.357+0000 7f8c2492e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.357+0000 7f8c2492e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2010a530 con 0x7f8c20106a70 2024-03-21T13:07:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.357+0000 7f8c1e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.357+0000 7f8c1e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.152:57956/0 (socket says 172.21.15.152:57956) 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.357+0000 7f8c1e59c700 1 -- 172.21.15.152:0/2254806662 learned_addr learned my addr 172.21.15.152:0/2254806662 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.357+0000 7f8c1e59c700 1 -- 172.21.15.152:0/2254806662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2006af20 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.357+0000 7f8c1e59c700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20197b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000f690 tx=0x7f8c1000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.358+0000 7f8c17fff700 1 -- 172.21.15.152:0/2254806662 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c10027070 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.358+0000 7f8c17fff700 1 -- 172.21.15.152:0/2254806662 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8c1000fb90 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.358+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c20198120 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.358+0000 7f8c17fff700 1 -- 172.21.15.152:0/2254806662 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c10017440 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.358+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c201985e0 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.359+0000 7f8c17fff700 1 -- 172.21.15.152:0/2254806662 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f8c1002a070 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.359+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c201021b0 con 0x7f8c20106a70 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.359+0000 7f8c17fff700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8c0c039220 0x7f8c0c03b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.359+0000 7f8c17fff700 1 -- 172.21.15.152:0/2254806662 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8c1001e030 con 0x7f8c20106a70 2024-03-21T13:07:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.359+0000 7f8c15d9b700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8c0c039220 0x7f8c0c03b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.360+0000 7f8c15d9b700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8c0c039220 0x7f8c0c03b6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8c08006fd0 tx=0x7f8c08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.365+0000 7f8c17fff700 1 -- 172.21.15.152:0/2254806662 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8c10013070 con 0x7f8c20106a70 2024-03-21T13:07:17.645 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.644+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c200637c0 con 0x7f8c20106a70 2024-03-21T13:07:17.645 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.645+0000 7f8c17fff700 1 -- 172.21.15.152:0/2254806662 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f8c10031020 con 0x7f8c20106a70 2024-03-21T13:07:17.646 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:07:17.646 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"9c355834-e783-11ee-95cd-87774f69a715","modified":"2024-03-21T13:05:37.944522Z","created":"2024-03-21T13:05:37.944522Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8c0c039220 msgr2=0x7f8c0c03b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8c0c039220 0x7f8c0c03b6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8c08006fd0 tx=0x7f8c08006e40 comp rx=0 tx=0).stop 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 msgr2=0x7f8c20197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20197b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000f690 tx=0x7f8c1000f770 comp rx=0 tx=0).stop 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 shutdown_connections 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8c0c039220 0x7f8c0c03b6e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 --2- 172.21.15.152:0/2254806662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c20106a70 0x7f8c20197b20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 >> 172.21.15.152:0/2254806662 conn(0x7f8c200fc580 msgr2=0x7f8c201088a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 shutdown_connections 2024-03-21T13:07:17.648 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:17.647+0000 7f8c2492e700 1 -- 172.21.15.152:0/2254806662 wait complete. 2024-03-21T13:07:17.649 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-21T13:07:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:17 smithi089 ceph-mon[30095]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/2254806662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:19.081 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 podman[34013]: 2024-03-21 13:07:18.96471585 +0000 UTC m=+0.017590990 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:07:19.081 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 podman[34013]: 2024-03-21 13:07:19.08133641 +0000 UTC m=+0.134211535 container create b11d157d3e17d970cd9fd5fecbdd56fc20d939f109410ca081081eba9884cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-21T13:07:19.519 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T13:07:19.519 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T13:07:19.545 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 podman[34013]: 2024-03-21 13:07:19.242044366 +0000 UTC m=+0.294919493 container init b11d157d3e17d970cd9fd5fecbdd56fc20d939f109410ca081081eba9884cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-21T13:07:19.545 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 podman[34013]: 2024-03-21 13:07:19.247703915 +0000 UTC m=+0.300579050 container start b11d157d3e17d970cd9fd5fecbdd56fc20d939f109410ca081081eba9884cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T13:07:19.545 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T13:07:19.545 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 7 2024-03-21T13:07:19.545 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: pidfile_write: ignore empty --pid-file 2024-03-21T13:07:19.545 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: load: jerasure load: lrc 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: RocksDB version: 7.9.2 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Git sha 0 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: DB SUMMARY 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: DB Session ID: ZQINLC4IH2AZEKCTTIP4 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: CURRENT file: CURRENT 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.error_if_exists: 0 2024-03-21T13:07:19.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.create_if_missing: 0 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.paranoid_checks: 1 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.env: 0x5649ca257b20 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.info_log: 0x5649cbbb9920 2024-03-21T13:07:19.547 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.statistics: (nil) 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.use_fsync: 0 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_log_file_size: 0 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.allow_fallocate: 1 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T13:07:19.549 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.use_direct_reads: 0 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.db_log_dir: 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.wal_dir: 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T13:07:19.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.write_buffer_manager: 0x5649cbb8a1e0 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.unordered_write: 0 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T13:07:19.551 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.row_cache: None 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.wal_filter: None 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.two_write_queues: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.wal_compression: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.atomic_flush: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T13:07:19.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.log_readahead_size: 0 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_background_jobs: 2 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_background_compactions: -1 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_subcompactions: 1 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T13:07:19.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_open_files: -1 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_background_flushes: -1 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Compression algorithms supported: 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kZSTD supported: 0 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kXpressCompression supported: 0 2024-03-21T13:07:19.554 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kZlibCompression supported: 1 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kSnappyCompression supported: 1 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kLZ4Compression supported: 1 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: kBZip2Compression supported: 0 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.merge_operator: 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_filter: None 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T13:07:19.555 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5649cbbb9a20) 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: cache_index_and_filter_blocks: 1 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: pin_top_level_index_and_filter: 1 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: index_type: 0 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: data_block_index_type: 0 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: index_shortening: 1 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: checksum: 4 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: no_block_cache: 0 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_cache: 0x5649cbbbd090 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_cache_name: BinnedLRUCache 2024-03-21T13:07:19.556 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_cache_options: 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: capacity : 536870912 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: num_shard_bits : 4 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: strict_capacity_limit : 0 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: high_pri_pool_ratio: 0.000 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_cache_compressed: (nil) 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: persistent_cache: (nil) 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_size: 4096 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_size_deviation: 10 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_restart_interval: 16 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: index_block_restart_interval: 1 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: metadata_block_size: 4096 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: partition_filters: 0 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: use_delta_encoding: 1 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: filter_policy: bloomfilter 2024-03-21T13:07:19.557 INFO:journalctl@ceph.mon.b.smithi152.stdout: whole_key_filtering: 1 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: verify_compression: 0 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: read_amp_bytes_per_bit: 0 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: format_version: 5 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: enable_index_compression: 1 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: block_align: 0 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: max_auto_readahead_size: 262144 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: prepopulate_block_cache: 0 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: initial_auto_readahead_size: 8192 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression: NoCompression 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T13:07:19.558 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.num_levels: 7 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:07:19.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T13:07:19.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T13:07:19.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T13:07:19.562 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.table_properties_collectors: 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.inplace_update_support: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.bloom_locality: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.max_successive_merges: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.ttl: 2592000 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T13:07:19.563 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.enable_blob_files: false 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.min_blob_size: 0 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-21T13:07:19.564 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dd33d36e-46d7-425c-a0a1-b84245e36bbf 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026439277477, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026439277898, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711026439, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dd33d36e-46d7-425c-a0a1-b84245e36bbf", "db_session_id": "ZQINLC4IH2AZEKCTTIP4", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026439277989, "job": 1, "event": "recovery_finished"} 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5649cbcac000 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: DB pointer 0x5649cbc96000 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout: ** DB Stats ** 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:07:19.565 INFO:journalctl@ceph.mon.b.smithi152.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:19.566 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T13:07:19.567 INFO:journalctl@ceph.mon.b.smithi152.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: Block cache BinnedLRUCache@0x5649cbbbd090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: using public_addr v2:172.21.15.152:0/0 -> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: starting mon.b rank -1 at public addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] at bind addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(???) e0 preinit fsid 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).mds e1 new map 2024-03-21T13:07:19.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout: e1 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout: legacy client fscid: -1 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout: No filesystems configured 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T13:07:19.569 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/762272355' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/762272355' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b;count:3 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: Deploying daemon mon.b on smithi152 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/2254806662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 ceph-mon[34051]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 bash[34013]: b11d157d3e17d970cd9fd5fecbdd56fc20d939f109410ca081081eba9884cc2d 2024-03-21T13:07:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:19 smithi152 systemd[1]: Started Ceph mon.b for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:07:19.756 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:07:24.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.354+0000 7f056e53c700 1 -- 172.21.15.152:0/529567771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 msgr2=0x7f0550005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:24.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.354+0000 7f056e53c700 1 --2- 172.21.15.152:0/529567771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 0x7f0550005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f056806b5a0 tx=0x7f055c00de60 comp rx=0 tx=0).stop 2024-03-21T13:07:24.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.354+0000 7f056e53c700 1 -- 172.21.15.152:0/529567771 shutdown_connections 2024-03-21T13:07:24.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.354+0000 7f056e53c700 1 --2- 172.21.15.152:0/529567771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 0x7f0550005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:24.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.354+0000 7f056e53c700 1 -- 172.21.15.152:0/529567771 >> 172.21.15.152:0/529567771 conn(0x7f0568077180 msgr2=0x7f0568077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:24.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.354+0000 7f056e53c700 1 -- 172.21.15.152:0/529567771 shutdown_connections 2024-03-21T13:07:24.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.354+0000 7f056e53c700 1 -- 172.21.15.152:0/529567771 wait complete. 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.355+0000 7f056e53c700 1 Processor -- start 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.355+0000 7f056e53c700 1 -- start start 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f056e53c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 0x7f0568198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f056e53c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 0x7f056819d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f056e53c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0568105d60 con 0x7f05500035c0 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f056e53c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0568105be0 con 0x7f05681994f0 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f05677fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 0x7f056819d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:24.359 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f05677fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 0x7f056819d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:59002/0 (socket says 172.21.15.152:59002) 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f05677fe700 1 -- 172.21.15.152:0/1448493518 learned_addr learned my addr 172.21.15.152:0/1448493518 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f0567fff700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 0x7f0568198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.356+0000 7f05677fe700 1 -- 172.21.15.152:0/1448493518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 msgr2=0x7f056819d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f05677fe700 1 -- 172.21.15.152:0/1448493518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 msgr2=0x7f056819d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f05677fe700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 0x7f056819d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f05677fe700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 0x7f056819d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f0567fff700 1 -- 172.21.15.152:0/1448493518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 msgr2=0x7f056819d830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:07:24.360 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f0567fff700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 0x7f056819d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f0567fff700 1 -- 172.21.15.152:0/1448493518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0568199b30 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f0567fff700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 0x7f0568198fb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f055c01b000 tx=0x7f055c003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f05657fa700 1 -- 172.21.15.152:0/1448493518 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f055c0056e0 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f05657fa700 1 -- 172.21.15.152:0/1448493518 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f055c029070 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056819dd70 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f05657fa700 1 -- 172.21.15.152:0/1448493518 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f055c024e10 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.357+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056819e2d0 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.358+0000 7f05657fa700 1 -- 172.21.15.152:0/1448493518 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f055c019520 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.359+0000 7f05657fa700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0554038fc0 0x7f055403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.359+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0568101a60 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.359+0000 7f05657fa700 1 -- 172.21.15.152:0/1448493518 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f055c04ebc0 con 0x7f05500035c0 2024-03-21T13:07:24.361 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.359+0000 7f05677fe700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0554038fc0 0x7f055403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:24.362 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.359+0000 7f05677fe700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0554038fc0 0x7f055403b480 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0558009180 tx=0x7f0558008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:24.367 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.367+0000 7f05657fa700 1 -- 172.21.15.152:0/1448493518 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f055c049070 con 0x7f05500035c0 2024-03-21T13:07:24.649 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.649+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f056819e650 con 0x7f05500035c0 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: mon.a calling monitor election 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: mon.b calling monitor election 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:24.650 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: monmap e2: 2 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: fsmap 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: mgrmap e13: y(active, since 41s) 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: overall HEALTH_OK 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:24.651 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:24.651 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.649+0000 7f05657fa700 1 -- 172.21.15.152:0/1448493518 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f055c00ac50 con 0x7f05500035c0 2024-03-21T13:07:24.652 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:07:24.652 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":2,"fsid":"9c355834-e783-11ee-95cd-87774f69a715","modified":"2024-03-21T13:07:19.299873Z","created":"2024-03-21T13:05:37.944522Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T13:07:24.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0554038fc0 msgr2=0x7f055403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:24.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0554038fc0 0x7f055403b480 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0558009180 tx=0x7f0558008040 comp rx=0 tx=0).stop 2024-03-21T13:07:24.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 msgr2=0x7f0568198fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:24.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 0x7f0568198fb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f055c01b000 tx=0x7f055c003ea0 comp rx=0 tx=0).stop 2024-03-21T13:07:24.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 shutdown_connections 2024-03-21T13:07:24.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05500035c0 0x7f0568198fb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:24.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0554038fc0 0x7f055403b480 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:24.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.651+0000 7f056e53c700 1 --2- 172.21.15.152:0/1448493518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05681994f0 0x7f056819d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:24.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.652+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 >> 172.21.15.152:0/1448493518 conn(0x7f0568077180 msgr2=0x7f05681067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:24.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.652+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 shutdown_connections 2024-03-21T13:07:24.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:24.652+0000 7f056e53c700 1 -- 172.21.15.152:0/1448493518 wait complete. 2024-03-21T13:07:24.654 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 2 2024-03-21T13:07:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:07:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: mon.a calling monitor election 2024-03-21T13:07:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:24.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: mon.b calling monitor election 2024-03-21T13:07:24.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:24.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:24.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T13:07:24.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: monmap e2: 2 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:07:24.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: fsmap 2024-03-21T13:07:24.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:24.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: mgrmap e13: y(active, since 41s) 2024-03-21T13:07:24.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: overall HEALTH_OK 2024-03-21T13:07:24.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:24.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:25 smithi152 ceph-mon[34051]: Deploying daemon mon.c on smithi089 2024-03-21T13:07:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/1448493518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:25 smithi152 ceph-mon[34051]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:25 smithi089 ceph-mon[30095]: Deploying daemon mon.c on smithi089 2024-03-21T13:07:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/1448493518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:25 smithi089 ceph-mon[30095]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:26.224 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T13:07:26.224 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T13:07:26.471 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:07:27.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 podman[39171]: 2024-03-21 13:07:27.400250593 +0000 UTC m=+0.018186639 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 podman[39171]: 2024-03-21 13:07:27.598955606 +0000 UTC m=+0.216891652 container create a1d69bee49df90eb3ed93ca944d37bfd2bb82acc021f4348493bd5b01b204904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 podman[39171]: 2024-03-21 13:07:27.801417652 +0000 UTC m=+0.419353698 container init a1d69bee49df90eb3ed93ca944d37bfd2bb82acc021f4348493bd5b01b204904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 podman[39171]: 2024-03-21 13:07:27.807191001 +0000 UTC m=+0.425127048 container start a1d69bee49df90eb3ed93ca944d37bfd2bb82acc021f4348493bd5b01b204904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 7 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pidfile_write: ignore empty --pid-file 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: load: jerasure load: lrc 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: RocksDB version: 7.9.2 2024-03-21T13:07:27.854 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Git sha 0 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: DB SUMMARY 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: DB Session ID: GZUQ091GLR6J79ICX3DH 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: CURRENT file: CURRENT 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.error_if_exists: 0 2024-03-21T13:07:27.855 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.create_if_missing: 0 2024-03-21T13:07:27.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.paranoid_checks: 1 2024-03-21T13:07:27.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T13:07:27.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T13:07:27.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.env: 0x555f5a75db20 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.info_log: 0x555f5b44d920 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.statistics: (nil) 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.use_fsync: 0 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_log_file_size: 0 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T13:07:27.858 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.allow_fallocate: 1 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.use_direct_reads: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.db_log_dir: 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.wal_dir: 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T13:07:27.859 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.write_buffer_manager: 0x555f5b41e1e0 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T13:07:27.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.unordered_write: 0 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.row_cache: None 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.wal_filter: None 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T13:07:27.861 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.two_write_queues: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.wal_compression: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.atomic_flush: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.log_readahead_size: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T13:07:27.862 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_background_jobs: 2 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_background_compactions: -1 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_subcompactions: 1 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T13:07:27.863 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_open_files: -1 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_background_flushes: -1 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Compression algorithms supported: 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kZSTD supported: 0 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kXpressCompression supported: 0 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kZlibCompression supported: 1 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kSnappyCompression supported: 1 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kLZ4Compression supported: 1 2024-03-21T13:07:27.864 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: kBZip2Compression supported: 0 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.merge_operator: 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_filter: None 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555f5b44da20) 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout: cache_index_and_filter_blocks: 1 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T13:07:27.865 INFO:journalctl@ceph.mon.c.smithi089.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: pin_top_level_index_and_filter: 1 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: index_type: 0 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: data_block_index_type: 0 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: index_shortening: 1 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: checksum: 4 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: no_block_cache: 0 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_cache: 0x555f5b451090 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_cache_name: BinnedLRUCache 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_cache_options: 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: capacity : 536870912 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: num_shard_bits : 4 2024-03-21T13:07:27.866 INFO:journalctl@ceph.mon.c.smithi089.stdout: strict_capacity_limit : 0 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: high_pri_pool_ratio: 0.000 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_cache_compressed: (nil) 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: persistent_cache: (nil) 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_size: 4096 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_size_deviation: 10 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_restart_interval: 16 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: index_block_restart_interval: 1 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: metadata_block_size: 4096 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: partition_filters: 0 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: use_delta_encoding: 1 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: filter_policy: bloomfilter 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: whole_key_filtering: 1 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: verify_compression: 0 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: read_amp_bytes_per_bit: 0 2024-03-21T13:07:27.867 INFO:journalctl@ceph.mon.c.smithi089.stdout: format_version: 5 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout: enable_index_compression: 1 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout: block_align: 0 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout: max_auto_readahead_size: 262144 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout: prepopulate_block_cache: 0 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout: initial_auto_readahead_size: 8192 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression: NoCompression 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.num_levels: 7 2024-03-21T13:07:27.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:07:27.869 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T13:07:27.870 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T13:07:27.871 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T13:07:27.872 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T13:07:27.873 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.table_properties_collectors: 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.inplace_update_support: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.bloom_locality: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.max_successive_merges: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T13:07:27.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.ttl: 2592000 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.enable_blob_files: false 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.min_blob_size: 0 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T13:07:27.875 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bfca40d4-1455-4389-8b8f-0fbd169e7e5e 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026447838023, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026447838417, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711026447, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bfca40d4-1455-4389-8b8f-0fbd169e7e5e", "db_session_id": "GZUQ091GLR6J79ICX3DH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711026447838499, "job": 1, "event": "recovery_finished"} 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555f5b540000 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: DB pointer 0x555f5b52a000 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout: ** DB Stats ** 2024-03-21T13:07:27.876 INFO:journalctl@ceph.mon.c.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: ** Compaction Stats [default] ** 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:07:27.877 INFO:journalctl@ceph.mon.c.smithi089.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: ** Compaction Stats [default] ** 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T13:07:27.878 INFO:journalctl@ceph.mon.c.smithi089.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T13:07:27.879 INFO:journalctl@ceph.mon.c.smithi089.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout: Block cache BinnedLRUCache@0x555f5b451090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout: 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-21T13:07:27.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: using public_addrv [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: starting mon.c rank -1 at public addrs [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] at bind addrs [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(???) e0 preinit fsid 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).mds e1 new map 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout: e1 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T13:07:27.881 INFO:journalctl@ceph.mon.c.smithi089.stdout: legacy client fscid: -1 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout: 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout: No filesystems configured 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:07:27.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/762272355' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.883 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/762272355' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.884 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:27.885 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:27.885 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:27.885 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi089:[v2:172.21.15.89:3301,v1:172.21.15.89:6790]=c;smithi152:172.21.15.152=b;count:3 2024-03-21T13:07:27.885 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: Deploying daemon mon.b on smithi152 2024-03-21T13:07:27.885 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.885 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/2254806662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.a calling monitor election 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.b calling monitor election 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:27.886 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: monmap e2: 2 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: fsmap 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mgrmap e13: y(active, since 41s) 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: overall HEALTH_OK 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: Deploying daemon mon.c on smithi089 2024-03-21T13:07:27.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/1448493518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:27.888 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:27.888 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:27.888 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 ceph-mon[39209]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T13:07:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 bash[39171]: a1d69bee49df90eb3ed93ca944d37bfd2bb82acc021f4348493bd5b01b204904 2024-03-21T13:07:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:27 smithi089 systemd[1]: Started Ceph mon.c for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:07:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: mon.a calling monitor election 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: mon.b calling monitor election 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: mon.c calling monitor election 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: mon.a calling monitor election 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0],c=[v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: fsmap 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: mgrmap e13: y(active, since 49s) 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: overall HEALTH_OK 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: mon.a calling monitor election 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: mon.b calling monitor election 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: mon.c calling monitor election 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: mon.a calling monitor election 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0],c=[v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: fsmap 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: mgrmap e13: y(active, since 49s) 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: overall HEALTH_OK 2024-03-21T13:07:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: mon.a calling monitor election 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: mon.b calling monitor election 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:33.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: mon.c calling monitor election 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: mon.a calling monitor election 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0],c=[v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: fsmap 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: mgrmap e13: y(active, since 49s) 2024-03-21T13:07:33.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: overall HEALTH_OK 2024-03-21T13:07:33.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:33.985 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.982+0000 7f2179f71700 1 -- 172.21.15.152:0/2926415675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2160005c90 msgr2=0x7f2160008120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.982+0000 7f2179f71700 1 --2- 172.21.15.152:0/2926415675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2160005c90 0x7f2160008120 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2174105300 tx=0x7f216400ff20 comp rx=0 tx=0).stop 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.982+0000 7f2179f71700 1 -- 172.21.15.152:0/2926415675 shutdown_connections 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.982+0000 7f2179f71700 1 --2- 172.21.15.152:0/2926415675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2160005c90 0x7f2160008120 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.982+0000 7f2179f71700 1 -- 172.21.15.152:0/2926415675 >> 172.21.15.152:0/2926415675 conn(0x7f21740ff960 msgr2=0x7f2174101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.982+0000 7f2179f71700 1 -- 172.21.15.152:0/2926415675 shutdown_connections 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.982+0000 7f2179f71700 1 -- 172.21.15.152:0/2926415675 wait complete. 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 Processor -- start 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 -- start start 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 0x7f21741990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 0x7f217419d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:33.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217419de80 0x7f217419e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217410b3c0 con 0x7f217419de80 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217410b240 con 0x7f2160005c90 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2179f71700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217410b540 con 0x7f2174199600 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f21737fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 0x7f21741990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f21737fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 0x7f21741990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:51606/0 (socket says 172.21.15.152:51606) 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f21737fe700 1 -- 172.21.15.152:0/1017391856 learned_addr learned my addr 172.21.15.152:0/1017391856 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f2173fff700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217419de80 0x7f217419e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f21737fe700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 msgr2=0x7f21741990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f21737fe700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 msgr2=0x7f21741990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f21737fe700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 0x7f21741990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f216bfff700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 0x7f217419d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:33.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f21737fe700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 0x7f21741990c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f216bfff700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 msgr2=0x7f217419d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f216bfff700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 msgr2=0x7f217419d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f216bfff700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 0x7f217419d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.983+0000 7f216bfff700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 0x7f217419d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2173fff700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 msgr2=0x7f217419d940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2173fff700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 0x7f217419d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2173fff700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 msgr2=0x7f21741990c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2173fff700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 0x7f21741990c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2173fff700 1 -- 172.21.15.152:0/1017391856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21741a5bf0 con 0x7f217419de80 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2173fff700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217419de80 0x7f217419e330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f216c00bd50 tx=0x7f216c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f21717fa700 1 -- 172.21.15.152:0/1017391856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c019070 con 0x7f217419de80 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f21717fa700 1 -- 172.21.15.152:0/1017391856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f216c00cd10 con 0x7f217419de80 2024-03-21T13:07:33.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f21717fa700 1 -- 172.21.15.152:0/1017391856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c0125a0 con 0x7f217419de80 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21741a5e20 con 0x7f217419de80 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.984+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741a6300 con 0x7f217419de80 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.985+0000 7f21717fa700 1 -- 172.21.15.152:0/1017391856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f216c01c070 con 0x7f217419de80 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.985+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21740637c0 con 0x7f217419de80 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.985+0000 7f21717fa700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f21540396b0 0x7f215403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.985+0000 7f21717fa700 1 -- 172.21.15.152:0/1017391856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f216c04bcb0 con 0x7f217419de80 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.985+0000 7f21737fe700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f21540396b0 0x7f215403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:33.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.986+0000 7f21737fe700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f21540396b0 0x7f215403bb70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2164005de0 tx=0x7f216400fa10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:33.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:33.990+0000 7f21717fa700 1 -- 172.21.15.152:0/1017391856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f216c0137a0 con 0x7f217419de80 2024-03-21T13:07:34.272 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.271+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21741a65c0 con 0x7f217419de80 2024-03-21T13:07:34.272 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:33 smithi152 ceph-mon[34051]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:34.272 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:34.272 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:34.272 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:34.272 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:34.273 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.272+0000 7f21717fa700 1 -- 172.21.15.152:0/1017391856 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1281 (secure 0 0 0) 0x7f216c01d3b0 con 0x7f217419de80 2024-03-21T13:07:34.273 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-21T13:07:34.273 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":3,"fsid":"9c355834-e783-11ee-95cd-87774f69a715","modified":"2024-03-21T13:07:27.858331Z","created":"2024-03-21T13:05:37.944522Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3301","nonce":0},{"type":"v1","addr":"172.21.15.89:6790","nonce":0}]},"addr":"172.21.15.89:6790/0","public_addr":"172.21.15.89:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f21540396b0 msgr2=0x7f215403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f21540396b0 0x7f215403bb70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2164005de0 tx=0x7f216400fa10 comp rx=0 tx=0).stop 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217419de80 msgr2=0x7f217419e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217419de80 0x7f217419e330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f216c00bd50 tx=0x7f216c00be30 comp rx=0 tx=0).stop 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 shutdown_connections 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217419de80 0x7f217419e330 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f21540396b0 0x7f215403bb70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:34.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2160005c90 0x7f21741990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:34.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 --2- 172.21.15.152:0/1017391856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2174199600 0x7f217419d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:34.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 >> 172.21.15.152:0/1017391856 conn(0x7f21740ff960 msgr2=0x7f2174107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:34.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 shutdown_connections 2024-03-21T13:07:34.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:34.274+0000 7f2179f71700 1 -- 172.21.15.152:0/1017391856 wait complete. 2024-03-21T13:07:34.276 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 3 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[39209]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[30095]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:34.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:34.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:34.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:07:34.965 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-21T13:07:34.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-21T13:07:35.213 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:35.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/1017391856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:35.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[30095]: Reconfiguring daemon mon.a on smithi089 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/1017391856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:35.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:35 smithi089 ceph-mon[39209]: Reconfiguring daemon mon.a on smithi089 2024-03-21T13:07:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/1017391856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:07:35.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:35 smithi152 ceph-mon[34051]: Reconfiguring daemon mon.a on smithi089 2024-03-21T13:07:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.244+0000 7f94b24e2700 1 -- 172.21.15.89:0/1535875488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac106bd0 msgr2=0x7f94ac106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.244+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1535875488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac106bd0 0x7f94ac106fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f949c009910 tx=0x7f949c009c20 comp rx=0 tx=0).stop 2024-03-21T13:07:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.244+0000 7f94b24e2700 1 -- 172.21.15.89:0/1535875488 shutdown_connections 2024-03-21T13:07:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.244+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1535875488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac106bd0 0x7f94ac106fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.244+0000 7f94b24e2700 1 -- 172.21.15.89:0/1535875488 >> 172.21.15.89:0/1535875488 conn(0x7f94ac0fbd10 msgr2=0x7f94ac0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.244+0000 7f94b24e2700 1 -- 172.21.15.89:0/1535875488 shutdown_connections 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.244+0000 7f94b24e2700 1 -- 172.21.15.89:0/1535875488 wait complete. 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.245+0000 7f94b24e2700 1 Processor -- start 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.245+0000 7f94b24e2700 1 -- start start 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b24e2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94ac106bd0 0x7f94ac1011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b24e2700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94ac101710 0x7f94ac101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b24e2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1a30d0 0x7f94ac1a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b24e2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac10b790 con 0x7f94ac1a30d0 2024-03-21T13:07:37.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b24e2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac10b610 con 0x7f94ac106bd0 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b24e2700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac10b910 con 0x7f94ac101710 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94ac101710 0x7f94ac101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94ac101710 0x7f94ac101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:48606/0 (socket says 172.21.15.89:48606) 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b0a7f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1a30d0 0x7f94ac1a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 -- 172.21.15.89:0/1701037614 learned_addr learned my addr 172.21.15.89:0/1701037614 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94abfff700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94ac106bd0 0x7f94ac1011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 -- 172.21.15.89:0/1701037614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94ac106bd0 msgr2=0x7f94ac1011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94ac106bd0 0x7f94ac1011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 -- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1a30d0 msgr2=0x7f94ac1a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1a30d0 0x7f94ac1a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 -- 172.21.15.89:0/1701037614 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac1a5b90 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94b0a7f700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1a30d0 0x7f94ac1a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.246+0000 7f94ab7fe700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94ac101710 0x7f94ac101b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f94a000b7b0 tx=0x7f94a000bac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.247+0000 7f94a97fa700 1 -- 172.21.15.89:0/1701037614 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a0004230 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.247+0000 7f94a97fa700 1 -- 172.21.15.89:0/1701037614 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f94a00043d0 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.247+0000 7f94a97fa700 1 -- 172.21.15.89:0/1701037614 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a0011a20 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.247+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac1a5e80 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.247+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac1a62e0 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.247+0000 7f94a97fa700 1 -- 172.21.15.89:0/1701037614 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f94a0011bc0 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.248+0000 7f94a97fa700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f94940396b0 0x7f949403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.248+0000 7f94a97fa700 1 -- 172.21.15.89:0/1701037614 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f94a004c000 con 0x7f94ac101710 2024-03-21T13:07:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.248+0000 7f94abfff700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f94940396b0 0x7f949403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:37.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.248+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac0637c0 con 0x7f94ac101710 2024-03-21T13:07:37.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.248+0000 7f94abfff700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f94940396b0 0x7f949403bb70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f949c005b40 tx=0x7f949c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:37.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.253+0000 7f94a97fa700 1 -- 172.21.15.89:0/1701037614 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94a0003bb0 con 0x7f94ac101710 2024-03-21T13:07:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.451+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f94ac1a6570 con 0x7f94ac101710 2024-03-21T13:07:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.451+0000 7f94a97fa700 1 -- 172.21.15.89:0/1701037614 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f94a0018510 con 0x7f94ac101710 2024-03-21T13:07:37.452 INFO:teuthology.orchestra.run.smithi089.stdout:# minimal ceph.conf for 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:07:37.452 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-03-21T13:07:37.452 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:07:37.452 INFO:teuthology.orchestra.run.smithi089.stdout: mon_host = [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] 2024-03-21T13:07:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f94940396b0 msgr2=0x7f949403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f94940396b0 0x7f949403bb70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f949c005b40 tx=0x7f949c01a040 comp rx=0 tx=0).stop 2024-03-21T13:07:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94ac101710 msgr2=0x7f94ac101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94ac101710 0x7f94ac101b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f94a000b7b0 tx=0x7f94a000bac0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 shutdown_connections 2024-03-21T13:07:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1a30d0 0x7f94ac1a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f94940396b0 0x7f949403bb70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94ac106bd0 0x7f94ac1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 --2- 172.21.15.89:0/1701037614 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94ac101710 0x7f94ac101b90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:37.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 >> 172.21.15.89:0/1701037614 conn(0x7f94ac0fbd10 msgr2=0x7f94ac103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:37.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 shutdown_connections 2024-03-21T13:07:37.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:37.453+0000 7f94b24e2700 1 -- 172.21.15.89:0/1701037614 wait complete. 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[30095]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1701037614' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:38.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[39209]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1701037614' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:37 smithi152 ceph-mon[34051]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1701037614' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:38.361 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-21T13:07:38.362 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:07:38.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T13:07:38.391 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:07:38.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:07:38.458 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:07:38.458 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T13:07:38.494 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:07:38.495 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:07:38.564 INFO:tasks.cephadm:Adding mgr.y on smithi089 2024-03-21T13:07:38.564 INFO:tasks.cephadm:Adding mgr.x on smithi152 2024-03-21T13:07:38.565 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi089=y;smithi152=x' 2024-03-21T13:07:38.841 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:07:39.063 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:38 smithi089 ceph-mon[30095]: Reconfiguring mon.c (monmap changed)... 2024-03-21T13:07:39.063 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:38 smithi089 ceph-mon[30095]: Reconfiguring daemon mon.c on smithi089 2024-03-21T13:07:39.063 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:38 smithi089 ceph-mon[39209]: Reconfiguring mon.c (monmap changed)... 2024-03-21T13:07:39.063 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:38 smithi089 ceph-mon[39209]: Reconfiguring daemon mon.c on smithi089 2024-03-21T13:07:39.292 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:38 smithi152 ceph-mon[34051]: Reconfiguring mon.c (monmap changed)... 2024-03-21T13:07:39.292 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:38 smithi152 ceph-mon[34051]: Reconfiguring daemon mon.c on smithi089 2024-03-21T13:07:40.240 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:39 smithi152 ceph-mon[34051]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:40.240 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:39 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:40.240 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:39 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:40.240 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:39 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:40.240 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:39 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:40.241 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:39 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[30095]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[39209]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:07:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:07:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:39 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:40.857 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.857+0000 7fdc0db05700 1 -- 172.21.15.152:0/1940723719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 msgr2=0x7fdc08100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:40.858 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.857+0000 7fdc0db05700 1 --2- 172.21.15.152:0/1940723719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 0x7fdc08100760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800aa20 tx=0x7fdbf800ad30 comp rx=0 tx=0).stop 2024-03-21T13:07:40.858 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.857+0000 7fdc0db05700 1 -- 172.21.15.152:0/1940723719 shutdown_connections 2024-03-21T13:07:40.858 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.857+0000 7fdc0db05700 1 --2- 172.21.15.152:0/1940723719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 0x7fdc08100760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:40.858 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.857+0000 7fdc0db05700 1 -- 172.21.15.152:0/1940723719 >> 172.21.15.152:0/1940723719 conn(0x7fdc080fb8e0 msgr2=0x7fdc080fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:40.858 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.857+0000 7fdc0db05700 1 -- 172.21.15.152:0/1940723719 shutdown_connections 2024-03-21T13:07:40.858 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.857+0000 7fdc0db05700 1 -- 172.21.15.152:0/1940723719 wait complete. 2024-03-21T13:07:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 Processor -- start 2024-03-21T13:07:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 -- start start 2024-03-21T13:07:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 0x7fdc080fda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdc080fc110 0x7fdc080fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc080fcad0 0x7fdc081a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc08104f60 con 0x7fdc08100340 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc08104de0 con 0x7fdc080fcad0 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc0db05700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081050e0 con 0x7fdc080fc110 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc07fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc080fcad0 0x7fdc081a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc077fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 0x7fdc080fda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc07fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc080fcad0 0x7fdc081a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:54962/0 (socket says 172.21.15.152:54962) 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc07fff700 1 -- 172.21.15.152:0/3023793596 learned_addr learned my addr 172.21.15.152:0/3023793596 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.858+0000 7fdc06ffd700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdc080fc110 0x7fdc080fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc077fe700 1 -- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdc080fc110 msgr2=0x7fdc080fc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc077fe700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdc080fc110 0x7fdc080fc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc077fe700 1 -- 172.21.15.152:0/3023793596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc080fcad0 msgr2=0x7fdc081a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc077fe700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc080fcad0 0x7fdc081a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc077fe700 1 -- 172.21.15.152:0/3023793596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc081a10d0 con 0x7fdc08100340 2024-03-21T13:07:40.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc077fe700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 0x7fdc080fda50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8006010 tx=0x7fdbf80050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:40.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc04ff9700 1 -- 172.21.15.152:0/3023793596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8025070 con 0x7fdc08100340 2024-03-21T13:07:40.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc04ff9700 1 -- 172.21.15.152:0/3023793596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdbf80054d0 con 0x7fdc08100340 2024-03-21T13:07:40.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc04ff9700 1 -- 172.21.15.152:0/3023793596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800e450 con 0x7fdc08100340 2024-03-21T13:07:40.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc081a1360 con 0x7fdc08100340 2024-03-21T13:07:40.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.859+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc081a1910 con 0x7fdc08100340 2024-03-21T13:07:40.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.860+0000 7fdc04ff9700 1 -- 172.21.15.152:0/3023793596 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7fdbf8028070 con 0x7fdc08100340 2024-03-21T13:07:40.863 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.860+0000 7fdc04ff9700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fdbf00396b0 0x7fdbf003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:40.863 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.860+0000 7fdc04ff9700 1 -- 172.21.15.152:0/3023793596 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdbf804e0c0 con 0x7fdc08100340 2024-03-21T13:07:40.863 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.860+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc080637c0 con 0x7fdc08100340 2024-03-21T13:07:40.863 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.860+0000 7fdc06ffd700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fdbf00396b0 0x7fdbf003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:40.863 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.861+0000 7fdc06ffd700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fdbf00396b0 0x7fdbf003bb70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009ad0 tx=0x7fdbfc009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:40.866 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:40.866+0000 7fdc04ff9700 1 -- 172.21.15.152:0/3023793596 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdbf802f070 con 0x7fdc08100340 2024-03-21T13:07:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:40 smithi152 ceph-mon[34051]: Reconfiguring mon.b (monmap changed)... 2024-03-21T13:07:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:40 smithi152 ceph-mon[34051]: Reconfiguring daemon mon.b on smithi152 2024-03-21T13:07:41.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.080+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi089=y;smithi152=x", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc080625d0 con 0x7fdbf00396b0 2024-03-21T13:07:41.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.085+0000 7fdc04ff9700 1 -- 172.21.15.152:0/3023793596 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdc080625d0 con 0x7fdbf00396b0 2024-03-21T13:07:41.086 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mgr update... 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fdbf00396b0 msgr2=0x7fdbf003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fdbf00396b0 0x7fdbf003bb70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009ad0 tx=0x7fdbfc009210 comp rx=0 tx=0).stop 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 msgr2=0x7fdc080fda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 0x7fdc080fda50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8006010 tx=0x7fdbf80050d0 comp rx=0 tx=0).stop 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 shutdown_connections 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fdbf00396b0 0x7fdbf003bb70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc080fcad0 0x7fdc081a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc08100340 0x7fdc080fda50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:41.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 --2- 172.21.15.152:0/3023793596 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdc080fc110 0x7fdc080fc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:41.090 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 >> 172.21.15.152:0/3023793596 conn(0x7fdc080fb8e0 msgr2=0x7fdc0818b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:41.090 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 shutdown_connections 2024-03-21T13:07:41.090 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:07:41.088+0000 7fdc0db05700 1 -- 172.21.15.152:0/3023793596 wait complete. 2024-03-21T13:07:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:40 smithi089 ceph-mon[30095]: Reconfiguring mon.b (monmap changed)... 2024-03-21T13:07:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:40 smithi089 ceph-mon[30095]: Reconfiguring daemon mon.b on smithi152 2024-03-21T13:07:41.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:40 smithi089 ceph-mon[39209]: Reconfiguring mon.b (monmap changed)... 2024-03-21T13:07:41.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:40 smithi089 ceph-mon[39209]: Reconfiguring daemon mon.b on smithi152 2024-03-21T13:07:41.851 DEBUG:teuthology.orchestra.run.smithi152:mgr.x> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.x.service 2024-03-21T13:07:41.853 INFO:tasks.cephadm:Deploying OSDs... 2024-03-21T13:07:41.853 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:07:41.853 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T13:07:41.876 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T13:07:41.876 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-03-21T13:07:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-21 13:07:27.225504161 +0000 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-21 13:01:53.811831130 +0000 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-21 13:01:53.811831130 +0000 2024-03-21T13:07:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-21 13:01:53.811831130 +0000 2024-03-21T13:07:41.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T13:07:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-21T13:07:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-21T13:07:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000278302 s, 1.8 MB/s 2024-03-21T13:07:42.008 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T13:07:42.066 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-03-21T13:07:42.122 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T13:07:42.122 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:42.122 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-21T13:07:42.122 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:42.122 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:42.123 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-21 13:07:27.225504161 +0000 2024-03-21T13:07:42.123 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-21 13:01:54.163823043 +0000 2024-03-21T13:07:42.123 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-21 13:01:54.163823043 +0000 2024-03-21T13:07:42.123 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-21 13:01:54.163823043 +0000 2024-03-21T13:07:42.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:42.154 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.155 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:07:42.155 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T13:07:42.155 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:07:42.155 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:42.189 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-21T13:07:42.190 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-21T13:07:42.190 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000208552 s, 2.5 MB/s 2024-03-21T13:07:42.191 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T13:07:42.247 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-03-21T13:07:42.302 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T13:07:42.302 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:42.303 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-21T13:07:42.303 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:42.303 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:42.303 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-21 13:07:27.226504138 +0000 2024-03-21T13:07:42.303 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-21 13:01:54.521814818 +0000 2024-03-21T13:07:42.303 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-21 13:01:54.521814818 +0000 2024-03-21T13:07:42.303 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-21 13:01:54.521814818 +0000 2024-03-21T13:07:42.303 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T13:07:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:42.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:42.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:07:42.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T13:07:42.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:07:42.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:42.352 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-21T13:07:42.352 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-21T13:07:42.352 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000205569 s, 2.5 MB/s 2024-03-21T13:07:42.353 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T13:07:42.409 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-03-21T13:07:42.464 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T13:07:42.464 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:42.464 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-21T13:07:42.464 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:42.464 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:42.465 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-21 13:07:27.226504138 +0000 2024-03-21T13:07:42.465 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-21 13:01:54.861807007 +0000 2024-03-21T13:07:42.465 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-21 13:01:54.861807007 +0000 2024-03-21T13:07:42.465 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-21 13:01:54.861807007 +0000 2024-03-21T13:07:42.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T13:07:42.530 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-21T13:07:42.530 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-21T13:07:42.530 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000195216 s, 2.6 MB/s 2024-03-21T13:07:42.531 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T13:07:42.587 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:07:42.587 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T13:07:42.602 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T13:07:42.603 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-21 13:07:18.760531321 +0000 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-21 13:01:47.044959623 +0000 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-21 13:01:47.044959623 +0000 2024-03-21T13:07:42.659 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-21 13:01:47.044959623 +0000 2024-03-21T13:07:42.659 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T13:07:42.725 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-21T13:07:42.725 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-21T13:07:42.726 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.00020726 s, 2.5 MB/s 2024-03-21T13:07:42.726 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T13:07:42.783 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-21 13:07:18.761531308 +0000 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-21 13:01:47.395955325 +0000 2024-03-21T13:07:42.838 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-21 13:01:47.395955325 +0000 2024-03-21T13:07:42.839 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-21 13:01:47.395955325 +0000 2024-03-21T13:07:42.839 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T13:07:42.905 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-21T13:07:42.905 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-21T13:07:42.905 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000193259 s, 2.6 MB/s 2024-03-21T13:07:42.906 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T13:07:42.962 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-03-21T13:07:43.021 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T13:07:43.021 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:43.021 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-21T13:07:43.022 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:43.022 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:43.022 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-21 13:07:18.761531308 +0000 2024-03-21T13:07:43.022 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-21 13:01:47.728951247 +0000 2024-03-21T13:07:43.022 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-21 13:01:47.728951247 +0000 2024-03-21T13:07:43.022 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-21 13:01:47.728951247 +0000 2024-03-21T13:07:43.022 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T13:07:43.122 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-21T13:07:43.122 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-21T13:07:43.122 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000184863 s, 2.8 MB/s 2024-03-21T13:07:43.123 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T13:07:43.139 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-03-21T13:07:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-21 13:07:43.077237470 +0000 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-21 13:01:48.051947292 +0000 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-21 13:01:48.051947292 +0000 2024-03-21T13:07:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-21 13:01:48.051947292 +0000 2024-03-21T13:07:43.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T13:07:43.262 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:42 smithi152 ceph-mon[34051]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi089=y;smithi152=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:43.262 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:42 smithi152 ceph-mon[34051]: Saving service mgr spec with placement smithi089=y;smithi152=x;count:2 2024-03-21T13:07:43.262 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:42 smithi152 ceph-mon[34051]: Deploying daemon mgr.x on smithi152 2024-03-21T13:07:43.301 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-21T13:07:43.301 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-21T13:07:43.301 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000181279 s, 2.8 MB/s 2024-03-21T13:07:43.302 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T13:07:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:42 smithi089 ceph-mon[30095]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi089=y;smithi152=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:42 smithi089 ceph-mon[30095]: Saving service mgr spec with placement smithi089=y;smithi152=x;count:2 2024-03-21T13:07:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:42 smithi089 ceph-mon[30095]: Deploying daemon mgr.x on smithi152 2024-03-21T13:07:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:42 smithi089 ceph-mon[39209]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi089=y;smithi152=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:42 smithi089 ceph-mon[39209]: Saving service mgr spec with placement smithi089=y;smithi152=x;count:2 2024-03-21T13:07:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:42 smithi089 ceph-mon[39209]: Deploying daemon mgr.x on smithi152 2024-03-21T13:07:43.359 INFO:tasks.cephadm:Deploying osd.0 on smithi089 with /dev/vg_nvme/lv_4... 2024-03-21T13:07:43.359 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T13:07:43.537 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 systemd[1]: Starting Ceph mgr.x for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:07:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:43.801 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 podman[36115]: 2024-03-21 13:07:43.439498399 +0000 UTC m=+0.017820240 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:07:43.801 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 podman[36115]: 2024-03-21 13:07:43.549829529 +0000 UTC m=+0.128151370 container create 6319992840fae99306b7d8429ec60d1038f592cdc6073596486019b2cb510db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:07:43.801 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 podman[36115]: 2024-03-21 13:07:43.771582327 +0000 UTC m=+0.349904174 container init 6319992840fae99306b7d8429ec60d1038f592cdc6073596486019b2cb510db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:07:43.801 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 podman[36115]: 2024-03-21 13:07:43.776848032 +0000 UTC m=+0.355169875 container start 6319992840fae99306b7d8429ec60d1038f592cdc6073596486019b2cb510db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-21T13:07:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:43 smithi152 ceph-mon[34051]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.072 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 ceph-mgr[36154]: -- 172.21.15.152:0/3784842745 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cb4a3885a0 con 0x55cb4a35cc00 2024-03-21T13:07:44.072 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 bash[36115]: 6319992840fae99306b7d8429ec60d1038f592cdc6073596486019b2cb510db0 2024-03-21T13:07:44.072 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:43 smithi152 systemd[1]: Started Ceph mgr.x for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:07:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[30095]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[39209]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:44.821 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:44 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:44.336+0000 7f36a460b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:07:44.821 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:44 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:44.432+0000 7f36a460b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:07:45.190 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:44 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:45.190 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:44 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:45.218 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:44 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:45.218 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:45.124+0000 7f36a460b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:07:45.218 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:45.217+0000 7f36a460b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:07:45.472 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:45.373+0000 7f36a460b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:07:45.472 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:45.471+0000 7f36a460b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:07:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T13:07:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T13:07:45.558 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-21T13:07:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-21T13:07:45.558 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112303 s, 934 MB/s 2024-03-21T13:07:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-21T13:07:45.818 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:45.554+0000 7f36a460b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:07:45.818 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:45.817+0000 7f36a460b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:07:46.071 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:45.928+0000 7f36a460b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:07:46.071 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:46.014+0000 7f36a460b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:07:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:07:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:46.144 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:46.144 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.144 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.144 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:46.145 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:46.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:46.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:07:46.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:07:46.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:46.172 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_4 2024-03-21T13:07:46.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:46.109+0000 7f36a460b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:07:46.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:46.287+0000 7f36a460b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:07:46.822 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:07:47.270 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:46 smithi152 ceph-mon[34051]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T13:07:47.270 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:46 smithi152 ceph-mon[34051]: Reconfiguring daemon mgr.y on smithi089 2024-03-21T13:07:47.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:46 smithi089 ceph-mon[30095]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T13:07:47.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:46 smithi089 ceph-mon[30095]: Reconfiguring daemon mgr.y on smithi089 2024-03-21T13:07:47.285 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:46 smithi089 ceph-mon[39209]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T13:07:47.285 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:46 smithi089 ceph-mon[39209]: Reconfiguring daemon mgr.y on smithi089 2024-03-21T13:07:47.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:47.269+0000 7f36a460b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:07:47.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:47.401+0000 7f36a460b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:07:47.999 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:47.735+0000 7f36a460b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:07:48.000 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:47.823+0000 7f36a460b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:07:48.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:47 smithi089 ceph-mon[30095]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:48.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:48.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.216 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[39209]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:48.216 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.217 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.217 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:48.217 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-mon[34051]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:48.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:48.323 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:48.023+0000 7f36a460b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:07:48.323 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:48.208+0000 7f36a460b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:07:48.758 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:48.497+0000 7f36a460b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:07:48.758 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:48.757+0000 7f36a460b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 -- 172.21.15.89:0/1046120071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 msgr2=0x7f65b410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/1046120071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 0x7f65b410a600 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00b0a0 tx=0x7f65ac00b3b0 comp rx=0 tx=0).stop 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 -- 172.21.15.89:0/1046120071 shutdown_connections 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/1046120071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 0x7f65b410a600 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/1046120071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b41021b0 0x7f65b41025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/1046120071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b41033b0 0x7f65b4103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 -- 172.21.15.89:0/1046120071 >> 172.21.15.89:0/1046120071 conn(0x7f65b40fd770 msgr2=0x7f65b40ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 -- 172.21.15.89:0/1046120071 shutdown_connections 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.973+0000 7f65ba3d5700 1 -- 172.21.15.89:0/1046120071 wait complete. 2024-03-21T13:07:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 Processor -- start 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 -- start start 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b41021b0 0x7f65b4197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b41033b0 0x7f65b4197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 0x7f65b419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b410d330 con 0x7f65b41021b0 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b410d1b0 con 0x7f65b41033b0 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65ba3d5700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b410d4b0 con 0x7f65b4103d50 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b8972700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 0x7f65b419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b41021b0 0x7f65b4197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b41021b0 0x7f65b4197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51230/0 (socket says 172.21.15.89:51230) 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b3fff700 1 -- 172.21.15.89:0/2079980096 learned_addr learned my addr 172.21.15.89:0/2079980096 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b37fe700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b41033b0 0x7f65b4197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b8972700 1 -- 172.21.15.89:0/2079980096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b41033b0 msgr2=0x7f65b4197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:48.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b8972700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b41033b0 0x7f65b4197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b8972700 1 -- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b41021b0 msgr2=0x7f65b4197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b8972700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b41021b0 0x7f65b4197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b8972700 1 -- 172.21.15.89:0/2079980096 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b419f570 con 0x7f65b4103d50 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.974+0000 7f65b8972700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 0x7f65b419eed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00b070 tx=0x7f65ac004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.975+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac01f030 con 0x7f65b4103d50 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.975+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b419f800 con 0x7f65b4103d50 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.975+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f65ac005040 con 0x7f65b4103d50 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.975+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac01b690 con 0x7f65b4103d50 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.975+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b419fcb0 con 0x7f65b4103d50 2024-03-21T13:07:48.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.975+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f65ac022030 con 0x7f65b4103d50 2024-03-21T13:07:48.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.976+0000 7f65b17fa700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f659c039460 0x7f659c03b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:07:48.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.976+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f65ac00f020 con 0x7f65b4103d50 2024-03-21T13:07:48.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.976+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4068e50 con 0x7f65b4103d50 2024-03-21T13:07:48.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.976+0000 7f65b3fff700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f659c039460 0x7f659c03b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:07:48.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.976+0000 7f65b3fff700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f659c039460 0x7f659c03b920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65a4005950 tx=0x7f65a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:07:48.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:48.982+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65ac029020 con 0x7f65b4103d50 2024-03-21T13:07:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:49 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:49 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:49 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:49.071 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:48.995+0000 7f36a460b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:07:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:49.186+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f65b402dbb0 con 0x7f659c039460 2024-03-21T13:07:49.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:49 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:49.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:49 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:49.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:49 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:49.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:49 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:49.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:49 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:49.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:49 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:07:50.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[30095]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:50.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:07:50.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:07:50.020 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:50.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[39209]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:50.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:07:50.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:07:50.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:50 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:50 smithi152 ceph-mon[34051]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:50 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:07:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:50 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:07:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:50 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:50.072 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:49.802+0000 7f36a460b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:07:50.697 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:50 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:50.378+0000 7f36a460b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:07:50.697 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:50 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:50.476+0000 7f36a460b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:07:50.697 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:07:50 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:07:50.696+0000 7f36a460b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:07:51.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:07:51.008+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mgrmap(e 14) v1 ==== 92074+0+0 (secure 0 0 0) 0x7f65ac009b80 con 0x7f65b4103d50 2024-03-21T13:07:51.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[30095]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:51.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[30095]: Standby manager daemon x started 2024-03-21T13:07:51.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:07:51.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:07:51.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:07:51.068 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:07:51.069 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[39209]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:51.069 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[39209]: Standby manager daemon x started 2024-03-21T13:07:51.069 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:07:51.069 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:07:51.069 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:07:51.069 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:51 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:07:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:51 smithi152 ceph-mon[34051]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:51 smithi152 ceph-mon[34051]: Standby manager daemon x started 2024-03-21T13:07:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:51 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:07:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:51 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:07:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:51 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:07:51.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:51 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/1130527787' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:07:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-21T13:07:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:07:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1577633017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]: dispatch 2024-03-21T13:07:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]: dispatch 2024-03-21T13:07:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]': finished 2024-03-21T13:07:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T13:07:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:52 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1577633017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]: dispatch 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]: dispatch 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]': finished 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T13:07:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1577633017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]: dispatch 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]: dispatch 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4258f99e-44cf-4cd0-a093-00e1192a8ef0"}]': finished 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T13:07:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:52 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/888786060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:07:53.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/888786060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:07:53.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/888786060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:07:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:54 smithi152 ceph-mon[34051]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:54.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:54 smithi089 ceph-mon[30095]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:54 smithi089 ceph-mon[39209]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:56.269 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:56 smithi089 ceph-mon[30095]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:56.269 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:56 smithi089 ceph-mon[39209]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:56 smithi152 ceph-mon[34051]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:58.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:58 smithi089 ceph-mon[30095]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:58.271 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:58 smithi089 ceph-mon[39209]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:58.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:58 smithi152 ceph-mon[34051]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:07:59.139 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T13:07:59.139 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:07:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:59.140 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:59 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T13:07:59.140 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:07:59 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:59 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T13:07:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:07:59 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:00.126 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:00 smithi089 ceph-mon[30095]: Deploying daemon osd.0 on smithi089 2024-03-21T13:08:00.126 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:00 smithi089 ceph-mon[30095]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:00.126 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:00 smithi089 ceph-mon[39209]: Deploying daemon osd.0 on smithi089 2024-03-21T13:08:00.126 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:00 smithi089 ceph-mon[39209]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:00 smithi152 ceph-mon[34051]: Deploying daemon osd.0 on smithi089 2024-03-21T13:08:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:00 smithi152 ceph-mon[34051]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:01.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:01 smithi152 ceph-mon[34051]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:01.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:01 smithi089 ceph-mon[30095]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:01.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:01 smithi089 ceph-mon[39209]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:04.229 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[30095]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:04.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:04.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:04.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:04.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[39209]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:04.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:04.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:04.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:03 smithi152 ceph-mon[34051]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:05 smithi152 ceph-mon[34051]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:05 smithi089 ceph-mon[30095]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:05 smithi089 ceph-mon[39209]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.836+0000 7f65b17fa700 1 -- 172.21.15.89:0/2079980096 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f65b402dbb0 con 0x7f659c039460 2024-03-21T13:08:06.838 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 0 on host 'smithi089' 2024-03-21T13:08:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f659c039460 msgr2=0x7f659c03b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f659c039460 0x7f659c03b920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65a4005950 tx=0x7f65a4009260 comp rx=0 tx=0).stop 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 msgr2=0x7f65b419eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 0x7f65b419eed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00b070 tx=0x7f65ac004c80 comp rx=0 tx=0).stop 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 shutdown_connections 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65b4103d50 0x7f65b419eed0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b41021b0 0x7f65b4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f659c039460 0x7f659c03b920 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 --2- 172.21.15.89:0/2079980096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b41033b0 0x7f65b4197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 >> 172.21.15.89:0/2079980096 conn(0x7f65b40fd770 msgr2=0x7f65b4106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 shutdown_connections 2024-03-21T13:08:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:06.839+0000 7f65ba3d5700 1 -- 172.21.15.89:0/2079980096 wait complete. 2024-03-21T13:08:07.502 DEBUG:teuthology.orchestra.run.smithi089:osd.0> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.0.service 2024-03-21T13:08:07.505 INFO:tasks.cephadm:Deploying osd.1 on smithi089 with /dev/vg_nvme/lv_3... 2024-03-21T13:08:07.505 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T13:08:08.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:07 smithi152 ceph-mon[34051]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:08:08.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:07 smithi152 ceph-mon[34051]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:08.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[30095]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:08:08.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[30095]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:08.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[39209]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:08:08.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[39209]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:08.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:08.193 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:08.905 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[30095]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T13:08:08.906 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[30095]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T13:08:08.906 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[30095]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:08.906 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:08.906 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[39209]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T13:08:08.906 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[39209]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T13:08:08.906 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[39209]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:08.906 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:08 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:08 smithi152 ceph-mon[34051]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T13:08:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:08 smithi152 ceph-mon[34051]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T13:08:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:08 smithi152 ceph-mon[34051]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:08 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:09.539 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0[43618]: 2024-03-21T13:08:09.259+0000 7fd8e6761700 -1 osd.0 0 waiting for initial osdmap 2024-03-21T13:08:09.539 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0[43618]: 2024-03-21T13:08:09.263+0000 7fd8de54c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:08:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:08:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:08:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:10.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T13:08:10.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:10.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' 2024-03-21T13:08:10.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' 2024-03-21T13:08:10.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:09 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:08:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:08:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T13:08:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:08:10.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: from='osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714]' entity='osd.0' 2024-03-21T13:08:10.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:09 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:10.590 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T13:08:10.591 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T13:08:10.591 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-21T13:08:10.591 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-21T13:08:10.591 INFO:teuthology.orchestra.run.smithi089.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138579 s, 757 MB/s 2024-03-21T13:08:10.591 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-21T13:08:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[30095]: osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] boot 2024-03-21T13:08:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[30095]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T13:08:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[30095]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:11.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[39209]: osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] boot 2024-03-21T13:08:11.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[39209]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T13:08:11.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:11 smithi089 ceph-mon[39209]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:11 smithi152 ceph-mon[34051]: osd.0 [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] boot 2024-03-21T13:08:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:11 smithi152 ceph-mon[34051]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T13:08:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:11 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:08:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:11 smithi152 ceph-mon[34051]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:11.736 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_3 2024-03-21T13:08:11.971 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:12.380 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi089 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi089 to 16064M 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:12.381 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi089 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi089 to 16064M 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:12.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:12.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:12.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T13:08:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi089 2024-03-21T13:08:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi089 to 16064M 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:13.378 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:13 smithi089 ceph-mon[30095]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:13.378 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:13 smithi089 ceph-mon[39209]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:13 smithi152 ceph-mon[34051]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:13.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.617+0000 7f2299d1a700 1 -- 172.21.15.89:0/925864614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294102cc0 msgr2=0x7f22941050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:13.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.617+0000 7f2299d1a700 1 --2- 172.21.15.89:0/925864614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294102cc0 0x7f22941050b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f22840098f0 tx=0x7f2284009c00 comp rx=0 tx=0).stop 2024-03-21T13:08:13.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.617+0000 7f2299d1a700 1 -- 172.21.15.89:0/925864614 shutdown_connections 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.617+0000 7f2299d1a700 1 --2- 172.21.15.89:0/925864614 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294107f20 0x7f229410a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.617+0000 7f2299d1a700 1 --2- 172.21.15.89:0/925864614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294102cc0 0x7f22941050b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.617+0000 7f2299d1a700 1 --2- 172.21.15.89:0/925864614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22941055f0 0x7f22941079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.617+0000 7f2299d1a700 1 -- 172.21.15.89:0/925864614 >> 172.21.15.89:0/925864614 conn(0x7f22940fc6f0 msgr2=0x7f22940feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 -- 172.21.15.89:0/925864614 shutdown_connections 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 -- 172.21.15.89:0/925864614 wait complete. 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 Processor -- start 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 -- start start 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294102cc0 0x7f22941025d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22941055f0 0x7f2294100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294107f20 0x7f22941011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229410d2b0 con 0x7f2294107f20 2024-03-21T13:08:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229410d130 con 0x7f22941055f0 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.618+0000 7f2299d1a700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229410d430 con 0x7f2294102cc0 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f22937fe700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294102cc0 0x7f22941025d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2293fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294107f20 0x7f22941011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f22937fe700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294102cc0 0x7f22941025d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:44748/0 (socket says 172.21.15.89:44748) 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f22937fe700 1 -- 172.21.15.89:0/841761117 learned_addr learned my addr 172.21.15.89:0/841761117 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2292ffd700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22941055f0 0x7f2294100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2293fff700 1 -- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294102cc0 msgr2=0x7f22941025d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2293fff700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294102cc0 0x7f22941025d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2293fff700 1 -- 172.21.15.89:0/841761117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22941055f0 msgr2=0x7f2294100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2293fff700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22941055f0 0x7f2294100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2293fff700 1 -- 172.21.15.89:0/841761117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2294101a80 con 0x7f2294107f20 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2292ffd700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22941055f0 0x7f2294100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f22937fe700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294102cc0 0x7f22941025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2293fff700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294107f20 0x7f22941011b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f228c009cd0 tx=0x7f228c00c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2290ff9700 1 -- 172.21.15.89:0/841761117 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c00cd90 con 0x7f2294107f20 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2290ff9700 1 -- 172.21.15.89:0/841761117 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f228c003a00 con 0x7f2294107f20 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22941a1b40 con 0x7f2294107f20 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2290ff9700 1 -- 172.21.15.89:0/841761117 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c004490 con 0x7f2294107f20 2024-03-21T13:08:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.619+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22941a2000 con 0x7f2294107f20 2024-03-21T13:08:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.620+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2294068e50 con 0x7f2294107f20 2024-03-21T13:08:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.621+0000 7f2290ff9700 1 -- 172.21.15.89:0/841761117 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 92074+0+0 (secure 0 0 0) 0x7f228c01a020 con 0x7f2294107f20 2024-03-21T13:08:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.621+0000 7f2290ff9700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f227c06e1c0 0x7f227c070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.621+0000 7f2290ff9700 1 -- 172.21.15.89:0/841761117 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f228c08ca10 con 0x7f2294107f20 2024-03-21T13:08:13.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.621+0000 7f22937fe700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f227c06e1c0 0x7f227c070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:13.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.622+0000 7f22937fe700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f227c06e1c0 0x7f227c070680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2284009680 tx=0x7f2284019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:08:13.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.626+0000 7f2290ff9700 1 -- 172.21.15.89:0/841761117 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f228c010040 con 0x7f2294107f20 2024-03-21T13:08:13.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:13.831+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f22940625d0 con 0x7f227c06e1c0 2024-03-21T13:08:14.537 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[39209]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:14.537 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:08:14.537 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:08:14.537 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:14.537 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[30095]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:14.537 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:08:14.538 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:08:14.538 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:14 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:14 smithi152 ceph-mon[34051]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:14 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:08:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:14 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:08:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:14 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:15.385 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:15 smithi089 ceph-mon[30095]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:15.385 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:15 smithi089 ceph-mon[39209]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:15.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:15 smithi152 ceph-mon[34051]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2447270083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "debf8a73-c369-49ed-8886-57f2ca61e6d5"}]: dispatch 2024-03-21T13:08:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2447270083' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "debf8a73-c369-49ed-8886-57f2ca61e6d5"}]': finished 2024-03-21T13:08:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:16 smithi152 ceph-mon[34051]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T13:08:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2447270083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "debf8a73-c369-49ed-8886-57f2ca61e6d5"}]: dispatch 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2447270083' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "debf8a73-c369-49ed-8886-57f2ca61e6d5"}]': finished 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[39209]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2447270083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "debf8a73-c369-49ed-8886-57f2ca61e6d5"}]: dispatch 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2447270083' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "debf8a73-c369-49ed-8886-57f2ca61e6d5"}]': finished 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[30095]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T13:08:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:17 smithi152 ceph-mon[34051]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4198346347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:08:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:17 smithi089 ceph-mon[30095]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4198346347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:08:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:17 smithi089 ceph-mon[39209]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4198346347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:08:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:19 smithi152 ceph-mon[34051]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:20.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:19 smithi089 ceph-mon[30095]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:19 smithi089 ceph-mon[39209]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:22.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:21 smithi089 ceph-mon[30095]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:21 smithi089 ceph-mon[39209]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:21 smithi152 ceph-mon[34051]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:22 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T13:08:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:22 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:22 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T13:08:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:22 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:22 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T13:08:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:22 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:24.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:23 smithi152 ceph-mon[34051]: Deploying daemon osd.1 on smithi089 2024-03-21T13:08:24.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:23 smithi152 ceph-mon[34051]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:23 smithi089 ceph-mon[30095]: Deploying daemon osd.1 on smithi089 2024-03-21T13:08:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:23 smithi089 ceph-mon[30095]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:23 smithi089 ceph-mon[39209]: Deploying daemon osd.1 on smithi089 2024-03-21T13:08:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:23 smithi089 ceph-mon[39209]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:26.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:25 smithi152 ceph-mon[34051]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:25 smithi089 ceph-mon[30095]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:26.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:25 smithi089 ceph-mon[39209]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:28.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:27 smithi089 ceph-mon[30095]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:28.181 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:27 smithi089 ceph-mon[39209]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:27 smithi152 ceph-mon[34051]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:29.703 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:29.704 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:29.704 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:29.704 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[30095]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:29.704 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:29.705 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:29.705 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:29.705 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:29 smithi089 ceph-mon[39209]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:29 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:29 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:29 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:29 smithi152 ceph-mon[34051]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:31.683 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 1 on host 'smithi089' 2024-03-21T13:08:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.682+0000 7f2290ff9700 1 -- 172.21.15.89:0/841761117 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f22940625d0 con 0x7f227c06e1c0 2024-03-21T13:08:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f227c06e1c0 msgr2=0x7f227c070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f227c06e1c0 0x7f227c070680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2284009680 tx=0x7f2284019040 comp rx=0 tx=0).stop 2024-03-21T13:08:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294107f20 msgr2=0x7f22941011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294107f20 0x7f22941011b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f228c009cd0 tx=0x7f228c00c620 comp rx=0 tx=0).stop 2024-03-21T13:08:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 shutdown_connections 2024-03-21T13:08:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2294107f20 0x7f22941011b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2294102cc0 0x7f22941025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f227c06e1c0 0x7f227c070680 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 --2- 172.21.15.89:0/841761117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22941055f0 0x7f2294100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 >> 172.21.15.89:0/841761117 conn(0x7f22940fc6f0 msgr2=0x7f2294103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:08:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 shutdown_connections 2024-03-21T13:08:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:31.684+0000 7f2299d1a700 1 -- 172.21.15.89:0/841761117 wait complete. 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[30095]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[30095]: from='osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[30095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[39209]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[39209]: from='osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[39209]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:08:32.201 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:32.202 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:31 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:32.228 DEBUG:teuthology.orchestra.run.smithi089:osd.1> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.1.service 2024-03-21T13:08:32.232 INFO:tasks.cephadm:Deploying osd.2 on smithi089 with /dev/vg_nvme/lv_2... 2024-03-21T13:08:32.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T13:08:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:31 smithi152 ceph-mon[34051]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:31 smithi152 ceph-mon[34051]: from='osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:08:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:31 smithi152 ceph-mon[34051]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:08:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:31 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:31 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:32.516 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:33.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[30095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[30095]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[30095]: from='osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[30095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[39209]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[39209]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:33.308 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[39209]: from='osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:33.309 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[39209]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:33.309 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:33.309 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:33 smithi152 ceph-mon[34051]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:08:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:33 smithi152 ceph-mon[34051]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T13:08:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:33 smithi152 ceph-mon[34051]: from='osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:33 smithi152 ceph-mon[34051]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:33.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:33.557 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1[49745]: 2024-03-21T13:08:33.431+0000 7f7ab0a01700 -1 osd.1 0 waiting for initial osdmap 2024-03-21T13:08:33.558 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:08:33 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1[49745]: 2024-03-21T13:08:33.433+0000 7f7aa87ec700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[39209]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[39209]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[39209]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[39209]: from='osd.1 ' entity='osd.1' 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[30095]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[30095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:34.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[30095]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T13:08:34.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:34.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:34.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:34 smithi089 ceph-mon[30095]: from='osd.1 ' entity='osd.1' 2024-03-21T13:08:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:34 smithi152 ceph-mon[34051]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:34 smithi152 ceph-mon[34051]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:34 smithi152 ceph-mon[34051]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T13:08:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:34 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:34.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:34 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:34.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:34 smithi152 ceph-mon[34051]: from='osd.1 ' entity='osd.1' 2024-03-21T13:08:34.926 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T13:08:34.926 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T13:08:34.926 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-21T13:08:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-21T13:08:34.927 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125442 s, 836 MB/s 2024-03-21T13:08:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-21T13:08:35.236 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:08:35.236 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:08:35.237 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:35.237 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[39209]: osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] boot 2024-03-21T13:08:35.237 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[39209]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T13:08:35.237 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:35.237 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:08:35.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:08:35.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:35.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[30095]: osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] boot 2024-03-21T13:08:35.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[30095]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T13:08:35.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:35 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:08:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:35 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:08:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:35.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:35 smithi152 ceph-mon[34051]: osd.1 [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] boot 2024-03-21T13:08:35.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:35 smithi152 ceph-mon[34051]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T13:08:35.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:08:35.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_2 2024-03-21T13:08:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:36 smithi152 ceph-mon[34051]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:36 smithi152 ceph-mon[34051]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T13:08:36.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:36 smithi089 ceph-mon[30095]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:36 smithi089 ceph-mon[30095]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T13:08:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:36 smithi089 ceph-mon[39209]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:08:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:36 smithi089 ceph-mon[39209]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T13:08:36.750 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:38.018 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi089 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi089 to 8032M 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:38.019 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:38.020 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:38.020 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.020 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[30095]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:38.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi089 2024-03-21T13:08:38.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.020 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:38.021 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:38.021 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi089 to 8032M 2024-03-21T13:08:38.021 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.021 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:38.021 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:38.021 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.022 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:38.022 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:38.022 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:38.022 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.022 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:37 smithi089 ceph-mon[39209]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi089 2024-03-21T13:08:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:08:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi089 to 8032M 2024-03-21T13:08:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:38.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:38.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:38.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:08:38.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:38.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:37 smithi152 ceph-mon[34051]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:38.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 -- 172.21.15.89:0/748574923 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 msgr2=0x7f96ac109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:38.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 --2- 172.21.15.89:0/748574923 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 0x7f96ac109430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f96a400b0a0 tx=0x7f96a400b3b0 comp rx=0 tx=0).stop 2024-03-21T13:08:38.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 -- 172.21.15.89:0/748574923 shutdown_connections 2024-03-21T13:08:38.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 --2- 172.21.15.89:0/748574923 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 0x7f96ac109430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:38.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 --2- 172.21.15.89:0/748574923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 0x7f96ac101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:38.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 --2- 172.21.15.89:0/748574923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96ac1021e0 0x7f96ac102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:38.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 -- 172.21.15.89:0/748574923 >> 172.21.15.89:0/748574923 conn(0x7f96ac0fc580 msgr2=0x7f96ac0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:08:38.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 -- 172.21.15.89:0/748574923 shutdown_connections 2024-03-21T13:08:38.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.369+0000 7f96b139f700 1 -- 172.21.15.89:0/748574923 wait complete. 2024-03-21T13:08:38.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 Processor -- start 2024-03-21T13:08:38.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 -- start start 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 0x7f96ac06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96ac1021e0 0x7f96ac19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 0x7f96ac1a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac10c120 con 0x7f96ac100fe0 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac10bfa0 con 0x7f96ac1021e0 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b139f700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac10c2a0 con 0x7f96ac102b80 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96abfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 0x7f96ac06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96abfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 0x7f96ac06a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55904/0 (socket says 172.21.15.89:55904) 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96b0b9e700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 0x7f96ac1a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96abfff700 1 -- 172.21.15.89:0/3300514785 learned_addr learned my addr 172.21.15.89:0/3300514785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.370+0000 7f96ab7fe700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96ac1021e0 0x7f96ac19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b0b9e700 1 -- 172.21.15.89:0/3300514785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96ac1021e0 msgr2=0x7f96ac19e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b0b9e700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96ac1021e0 0x7f96ac19e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b0b9e700 1 -- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 msgr2=0x7f96ac06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b0b9e700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 0x7f96ac06a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b0b9e700 1 -- 172.21.15.89:0/3300514785 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac1a1630 con 0x7f96ac102b80 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96ab7fe700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96ac1021e0 0x7f96ac19e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96abfff700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 0x7f96ac06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b0b9e700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 0x7f96ac1a0e50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009fd0 tx=0x7f96a4004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96a97fa700 1 -- 172.21.15.89:0/3300514785 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a401f030 con 0x7f96ac102b80 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96a97fa700 1 -- 172.21.15.89:0/3300514785 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f96a4005040 con 0x7f96ac102b80 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac1a18c0 con 0x7f96ac102b80 2024-03-21T13:08:38.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96a97fa700 1 -- 172.21.15.89:0/3300514785 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a401b690 con 0x7f96ac102b80 2024-03-21T13:08:38.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.371+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac1a1d10 con 0x7f96ac102b80 2024-03-21T13:08:38.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.372+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac068e50 con 0x7f96ac102b80 2024-03-21T13:08:38.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.372+0000 7f96a97fa700 1 -- 172.21.15.89:0/3300514785 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 14) v1 ==== 92074+0+0 (secure 0 0 0) 0x7f96a4022030 con 0x7f96ac102b80 2024-03-21T13:08:38.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.373+0000 7f96a97fa700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f969406e1c0 0x7f9694070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:08:38.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.373+0000 7f96abfff700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f969406e1c0 0x7f9694070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:08:38.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.373+0000 7f96a97fa700 1 -- 172.21.15.89:0/3300514785 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2199+0+0 (secure 0 0 0) 0x7f96a400f020 con 0x7f96ac102b80 2024-03-21T13:08:38.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.373+0000 7f96abfff700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f969406e1c0 0x7f9694070680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96a0005950 tx=0x7f96a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:08:38.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.378+0000 7f96a97fa700 1 -- 172.21.15.89:0/3300514785 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96a4029020 con 0x7f96ac102b80 2024-03-21T13:08:38.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:38.582+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f96ac02dcf0 con 0x7f969406e1c0 2024-03-21T13:08:39.172 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:38 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:08:39.172 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:38 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:08:39.172 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:38 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:39.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:38 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:08:39.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:38 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:08:39.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:38 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:38 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:08:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:38 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:08:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:38 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:39 smithi089 ceph-mon[30095]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:39 smithi089 ceph-mon[30095]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:39 smithi089 ceph-mon[39209]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:39 smithi089 ceph-mon[39209]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:39 smithi152 ceph-mon[34051]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:39 smithi152 ceph-mon[34051]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:41 smithi152 ceph-mon[34051]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/46396232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]: dispatch 2024-03-21T13:08:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]: dispatch 2024-03-21T13:08:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]': finished 2024-03-21T13:08:42.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:41 smithi152 ceph-mon[34051]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T13:08:42.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:42.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2980003973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[30095]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/46396232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]: dispatch 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]: dispatch 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]': finished 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[30095]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2980003973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:08:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[39209]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/46396232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]: dispatch 2024-03-21T13:08:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]: dispatch 2024-03-21T13:08:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2686770f-bbbc-4082-bb00-9f6f10f8783b"}]': finished 2024-03-21T13:08:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[39209]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T13:08:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2980003973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:08:44.235 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:43 smithi089 ceph-mon[30095]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:44.236 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:43 smithi089 ceph-mon[39209]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:43 smithi152 ceph-mon[34051]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:46.259 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:45 smithi089 ceph-mon[30095]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:46.260 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:45 smithi089 ceph-mon[39209]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:45 smithi152 ceph-mon[34051]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:48.292 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:47 smithi089 ceph-mon[30095]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:48.292 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T13:08:48.293 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:48.293 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:47 smithi089 ceph-mon[39209]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:48.293 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T13:08:48.293 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:47 smithi152 ceph-mon[34051]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T13:08:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:08:49.264 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:48 smithi089 ceph-mon[39209]: Deploying daemon osd.2 on smithi089 2024-03-21T13:08:49.265 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:48 smithi089 ceph-mon[30095]: Deploying daemon osd.2 on smithi089 2024-03-21T13:08:49.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:48 smithi152 ceph-mon[34051]: Deploying daemon osd.2 on smithi089 2024-03-21T13:08:50.204 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:49 smithi089 ceph-mon[30095]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:50.204 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:49 smithi089 ceph-mon[39209]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:50.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:49 smithi152 ceph-mon[34051]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:52.170 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:51 smithi089 ceph-mon[30095]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:52.171 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:51 smithi089 ceph-mon[39209]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:51 smithi152 ceph-mon[34051]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:54.272 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[39209]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:54.272 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:54.272 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:54.272 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:54.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[30095]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:54.273 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:54.273 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:54.273 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:53 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:53 smithi152 ceph-mon[34051]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:53 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:53 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:53 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:08:56.133 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:55 smithi089 ceph-mon[30095]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:56.134 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:55 smithi089 ceph-mon[39209]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:56.184 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 2 on host 'smithi089' 2024-03-21T13:08:56.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.182+0000 7f96a97fa700 1 -- 172.21.15.89:0/3300514785 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f96ac02dcf0 con 0x7f969406e1c0 2024-03-21T13:08:56.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f969406e1c0 msgr2=0x7f9694070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f969406e1c0 0x7f9694070680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96a0005950 tx=0x7f96a0009260 comp rx=0 tx=0).stop 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 msgr2=0x7f96ac1a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 0x7f96ac1a0e50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009fd0 tx=0x7f96a4004c80 comp rx=0 tx=0).stop 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 shutdown_connections 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96ac102b80 0x7f96ac1a0e50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96ac100fe0 0x7f96ac06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f969406e1c0 0x7f9694070680 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 --2- 172.21.15.89:0/3300514785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96ac1021e0 0x7f96ac19e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 >> 172.21.15.89:0/3300514785 conn(0x7f96ac0fc580 msgr2=0x7f96ac1054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 shutdown_connections 2024-03-21T13:08:56.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:08:56.185+0000 7f96b139f700 1 -- 172.21.15.89:0/3300514785 wait complete. 2024-03-21T13:08:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:55 smithi152 ceph-mon[34051]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:56.746 DEBUG:teuthology.orchestra.run.smithi089:osd.2> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.2.service 2024-03-21T13:08:56.748 INFO:tasks.cephadm:Deploying osd.3 on smithi089 with /dev/vg_nvme/lv_1... 2024-03-21T13:08:56.748 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T13:08:57.069 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[39209]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[30095]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:56 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:56 smithi152 ceph-mon[34051]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T13:08:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:56 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:56 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:56 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:56 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:08:58.201 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:58.202 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T13:08:58.202 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T13:08:58.202 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:58.202 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:58.202 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:57 smithi089 ceph-mon[30095]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:58.202 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:57 smithi089 ceph-mon[30095]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T13:08:58.202 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:57 smithi089 ceph-mon[30095]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T13:08:58.203 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:57 smithi089 ceph-mon[30095]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:58.203 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:57 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:58.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:57 smithi152 ceph-mon[34051]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:08:58.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:57 smithi152 ceph-mon[34051]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T13:08:58.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:57 smithi152 ceph-mon[34051]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T13:08:58.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:57 smithi152 ceph-mon[34051]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:08:58.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:57 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:58.498 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2[55730]: 2024-03-21T13:08:58.424+0000 7fbb1eeb7700 -1 osd.2 0 waiting for initial osdmap 2024-03-21T13:08:58.498 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2[55730]: 2024-03-21T13:08:58.427+0000 7fbb16ca2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:08:59.106 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T13:08:59.107 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T13:08:59.107 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-21T13:08:59.107 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-21T13:08:59.107 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123951 s, 846 MB/s 2024-03-21T13:08:59.107 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-21T13:08:59.175 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:59.175 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T13:08:59.175 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.175 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.175 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' 2024-03-21T13:08:59.175 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[30095]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:59.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[30095]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T13:08:59.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[30095]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' 2024-03-21T13:08:59.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:58 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:58 smithi152 ceph-mon[34051]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:08:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:58 smithi152 ceph-mon[34051]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T13:08:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:58 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:58 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:08:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:58 smithi152 ceph-mon[34051]: from='osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428]' entity='osd.2' 2024-03-21T13:08:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:58 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:09:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:59 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:09:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:59 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:09:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:59 smithi152 ceph-mon[34051]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:09:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:59 smithi152 ceph-mon[34051]: osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] boot 2024-03-21T13:09:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:59 smithi152 ceph-mon[34051]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T13:09:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:08:59 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[30095]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[30095]: osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] boot 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[30095]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[39209]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:09:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[39209]: osd.2 [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] boot 2024-03-21T13:09:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[39209]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T13:09:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:08:59 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:09:00.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_1 2024-03-21T13:09:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:09:02.228 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:01 smithi089 ceph-mon[30095]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:02.229 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:01 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:09:02.229 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:01 smithi089 ceph-mon[30095]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T13:09:02.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:01 smithi089 ceph-mon[39209]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:02.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:01 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:09:02.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:01 smithi089 ceph-mon[39209]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T13:09:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:01 smithi152 ceph-mon[34051]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:01 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:09:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:01 smithi152 ceph-mon[34051]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 -- 172.21.15.89:0/2097947426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50102cd0 msgr2=0x7efd501050c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 --2- 172.21.15.89:0/2097947426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50102cd0 0x7efd501050c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd40009990 tx=0x7efd40009ca0 comp rx=0 tx=0).stop 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 -- 172.21.15.89:0/2097947426 shutdown_connections 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 --2- 172.21.15.89:0/2097947426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50107f30 0x7efd5010a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 --2- 172.21.15.89:0/2097947426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50102cd0 0x7efd501050c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 --2- 172.21.15.89:0/2097947426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd50105600 0x7efd501079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 -- 172.21.15.89:0/2097947426 >> 172.21.15.89:0/2097947426 conn(0x7efd500fc660 msgr2=0x7efd500feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 -- 172.21.15.89:0/2097947426 shutdown_connections 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 -- 172.21.15.89:0/2097947426 wait complete. 2024-03-21T13:09:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.106+0000 7efd554e5700 1 Processor -- start 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd554e5700 1 -- start start 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd554e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50102cd0 0x7efd50199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd554e5700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50105600 0x7efd5019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd554e5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd50107f30 0x7efd501a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd554e5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5010d2a0 con 0x7efd50102cd0 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd554e5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5010d120 con 0x7efd50107f30 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd554e5700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5010d420 con 0x7efd50105600 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4e7fc700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50105600 0x7efd5019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50102cd0 0x7efd50199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4e7fc700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50105600 0x7efd5019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:42298/0 (socket says 172.21.15.89:42298) 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4e7fc700 1 -- 172.21.15.89:0/557830325 learned_addr learned my addr 172.21.15.89:0/557830325 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50102cd0 0x7efd50199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38348/0 (socket says 172.21.15.89:38348) 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4f7fe700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd50107f30 0x7efd501a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 -- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50105600 msgr2=0x7efd5019a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50105600 0x7efd5019a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 -- 172.21.15.89:0/557830325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd50107f30 msgr2=0x7efd501a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd50107f30 0x7efd501a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 -- 172.21.15.89:0/557830325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd501a17e0 con 0x7efd50102cd0 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4e7fc700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50105600 0x7efd5019a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.107+0000 7efd4effd700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50102cd0 0x7efd50199b80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efd400098e0 tx=0x7efd4000bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.108+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40026070 con 0x7efd50102cd0 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.108+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7efd40003d10 con 0x7efd50102cd0 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.108+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd501a1a70 con 0x7efd50102cd0 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.108+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4000fde0 con 0x7efd50102cd0 2024-03-21T13:09:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.108+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd501a1f40 con 0x7efd50102cd0 2024-03-21T13:09:03.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.109+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd50068e50 con 0x7efd50102cd0 2024-03-21T13:09:03.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.109+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 92074+0+0 (secure 0 0 0) 0x7efd40029070 con 0x7efd50102cd0 2024-03-21T13:09:03.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.110+0000 7efd37fff700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7efd3806e1c0 0x7efd38070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:03.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.110+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3017+0+0 (secure 0 0 0) 0x7efd4008e280 con 0x7efd50102cd0 2024-03-21T13:09:03.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.110+0000 7efd4e7fc700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7efd3806e1c0 0x7efd38070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:03.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.110+0000 7efd4e7fc700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7efd3806e1c0 0x7efd38070680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efd44009ea0 tx=0x7efd44009290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:09:03.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.115+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efd40013070 con 0x7efd50102cd0 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:03.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:03.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:03.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:03.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:03.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:03 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:03.321+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7efd500625d0 con 0x7efd3806e1c0 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:03.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:03 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:04.137 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:03 smithi089 sudo[60386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T13:09:04.137 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:03 smithi089 sudo[60386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:04.137 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:03 smithi089 sudo[60386]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:04.137 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:03 smithi089 sudo[60386]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi089 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi089 to 5354M 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:04.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[30095]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi089 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi089 to 5354M 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:04.139 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:04 smithi089 ceph-mon[39209]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:09:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi089 2024-03-21T13:09:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi089 to 5354M 2024-03-21T13:09:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:09:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T13:09:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:04 smithi152 ceph-mon[34051]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:09:04.405 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60408]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T13:09:04.405 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60408]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:04.406 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60408]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:04.406 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60408]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:04.461+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7efd40009d10 con 0x7efd50102cd0 2024-03-21T13:09:04.668 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60431]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T13:09:04.668 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60431]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:04.668 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60431]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:04.668 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60431]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:04.931 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60441]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T13:09:04.932 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60441]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:04.932 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60441]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:04.932 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60441]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:04.932 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60463]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T13:09:04.932 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:04.932 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60463]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:04.932 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:04 smithi089 sudo[60463]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:05.183 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[39209]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:05.183 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:05.183 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[39209]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T13:09:05.183 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[39209]: osdmap e22: 3 total, 3 up, 3 in 2024-03-21T13:09:05.184 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[30095]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:05.184 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:05.184 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[30095]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T13:09:05.184 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 ceph-mon[30095]: osdmap e22: 3 total, 3 up, 3 in 2024-03-21T13:09:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 ceph-mon[34051]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 ceph-mon[34051]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T13:09:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 ceph-mon[34051]: osdmap e22: 3 total, 3 up, 3 in 2024-03-21T13:09:05.446 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60497]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T13:09:05.446 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60497]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:05.446 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60497]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:05.446 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60497]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:05.446 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60487]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T13:09:05.447 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60487]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:05.447 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60487]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:05.447 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60487]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60525]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T13:09:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60525]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60525]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:05 smithi089 sudo[60525]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:06.195 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 sudo[36878]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T13:09:06.195 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 sudo[36878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:06.195 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 sudo[36878]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:06.195 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:05 smithi152 sudo[36878]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:06.467 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:06.467 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:06.467 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:06.467 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: Cluster is now healthy 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:06.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:09:06.469 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: Cluster is now healthy 2024-03-21T13:09:06.469 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:06.469 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:06.469 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:06.469 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:06.469 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:06.469 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:06 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 sudo[36881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 sudo[36881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 sudo[36881]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 sudo[36881]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: Cluster is now healthy 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:06 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:06.733 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60556]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T13:09:06.733 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60556]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:06.733 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60556]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:06.733 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60556]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:07.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60584]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T13:09:07.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60584]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T13:09:07.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60584]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T13:09:07.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:06 smithi089 sudo[60584]: pam_unix(sudo:session): session closed for user root 2024-03-21T13:09:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/732259394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c996867-ce38-4ae7-b934-5faf2ce2fc0e"}]: dispatch 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/732259394' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c996867-ce38-4ae7-b934-5faf2ce2fc0e"}]': finished 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:07.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:07.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3584047044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:07.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:07 smithi152 ceph-mon[34051]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/732259394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c996867-ce38-4ae7-b934-5faf2ce2fc0e"}]: dispatch 2024-03-21T13:09:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/732259394' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c996867-ce38-4ae7-b934-5faf2ce2fc0e"}]': finished 2024-03-21T13:09:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T13:09:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3584047044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:07.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[30095]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:07.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:07.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/732259394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c996867-ce38-4ae7-b934-5faf2ce2fc0e"}]: dispatch 2024-03-21T13:09:07.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:07.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/732259394' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c996867-ce38-4ae7-b934-5faf2ce2fc0e"}]': finished 2024-03-21T13:09:07.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3584047044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:07.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:07 smithi089 ceph-mon[39209]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:09 smithi089 ceph-mon[30095]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:10.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:09 smithi089 ceph-mon[39209]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:09 smithi152 ceph-mon[34051]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:12.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:11 smithi089 ceph-mon[30095]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 166 KiB/s wr, 11 op/s 2024-03-21T13:09:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:11 smithi089 ceph-mon[39209]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 166 KiB/s wr, 11 op/s 2024-03-21T13:09:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:11 smithi152 ceph-mon[34051]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 166 KiB/s wr, 11 op/s 2024-03-21T13:09:13.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T13:09:13.308 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:13.308 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T13:09:13.308 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T13:09:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:14.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:13 smithi089 ceph-mon[30095]: Deploying daemon osd.3 on smithi089 2024-03-21T13:09:14.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:13 smithi089 ceph-mon[30095]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 117 KiB/s wr, 8 op/s 2024-03-21T13:09:14.305 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:13 smithi089 ceph-mon[39209]: Deploying daemon osd.3 on smithi089 2024-03-21T13:09:14.305 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:13 smithi089 ceph-mon[39209]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 117 KiB/s wr, 8 op/s 2024-03-21T13:09:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:13 smithi152 ceph-mon[34051]: Deploying daemon osd.3 on smithi089 2024-03-21T13:09:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:13 smithi152 ceph-mon[34051]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 117 KiB/s wr, 8 op/s 2024-03-21T13:09:16.150 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:15 smithi089 ceph-mon[30095]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 90 KiB/s wr, 6 op/s 2024-03-21T13:09:16.150 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:15 smithi089 ceph-mon[39209]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 90 KiB/s wr, 6 op/s 2024-03-21T13:09:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:15 smithi152 ceph-mon[34051]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 90 KiB/s wr, 6 op/s 2024-03-21T13:09:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:17 smithi152 ceph-mon[34051]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-21T13:09:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:18.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[30095]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-21T13:09:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[39209]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-21T13:09:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:19 smithi152 ceph-mon[34051]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T13:09:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:19 smithi089 ceph-mon[30095]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T13:09:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:19 smithi089 ceph-mon[39209]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T13:09:20.971 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 3 on host 'smithi089' 2024-03-21T13:09:20.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.970+0000 7efd37fff700 1 -- 172.21.15.89:0/557830325 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efd500625d0 con 0x7efd3806e1c0 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7efd3806e1c0 msgr2=0x7efd38070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7efd3806e1c0 0x7efd38070680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efd44009ea0 tx=0x7efd44009290 comp rx=0 tx=0).stop 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50102cd0 msgr2=0x7efd50199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50102cd0 0x7efd50199b80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efd400098e0 tx=0x7efd4000bb30 comp rx=0 tx=0).stop 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 shutdown_connections 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7efd3806e1c0 0x7efd38070680 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd50107f30 0x7efd501a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd50102cd0 0x7efd50199b80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 --2- 172.21.15.89:0/557830325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd50105600 0x7efd5019a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 >> 172.21.15.89:0/557830325 conn(0x7efd500fc660 msgr2=0x7efd50101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:09:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 shutdown_connections 2024-03-21T13:09:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:09:20.972+0000 7efd554e5700 1 -- 172.21.15.89:0/557830325 wait complete. 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[30095]: from='osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[30095]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[39209]: from='osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[39209]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:21.276 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:21 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:21 smithi152 ceph-mon[34051]: from='osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:09:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:21 smithi152 ceph-mon[34051]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:09:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:21 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:21 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:21.552 DEBUG:teuthology.orchestra.run.smithi089:osd.3> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.3.service 2024-03-21T13:09:21.554 INFO:tasks.cephadm:Deploying osd.4 on smithi152 with /dev/vg_nvme/lv_4... 2024-03-21T13:09:21.555 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T13:09:21.798 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:22.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T13:09:22.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T13:09:22.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T13:09:22.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: from='osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:09:22.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:22.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:09:22.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:22.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:22 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:22.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T13:09:22.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T13:09:22.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T13:09:22.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: from='osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:09:22.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:22.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:09:22.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: from='osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-21T13:09:22.295 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:22.297 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:22.545 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3[62387]: 2024-03-21T13:09:22.472+0000 7fc59a529700 -1 osd.3 0 waiting for initial osdmap 2024-03-21T13:09:22.545 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:09:22 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3[62387]: 2024-03-21T13:09:22.476+0000 7fc591b13700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:09:23.168 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[30095]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:09:23.168 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[30095]: osdmap e25: 4 total, 3 up, 4 in 2024-03-21T13:09:23.168 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[30095]: from='osd.3 ' entity='osd.3' 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[30095]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[39209]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[39209]: osdmap e25: 4 total, 3 up, 4 in 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[39209]: from='osd.3 ' entity='osd.3' 2024-03-21T13:09:23.169 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[39209]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:23.170 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:23 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:23 smithi152 ceph-mon[34051]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-21T13:09:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:23 smithi152 ceph-mon[34051]: osdmap e25: 4 total, 3 up, 4 in 2024-03-21T13:09:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:23 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:23 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:23 smithi152 ceph-mon[34051]: from='osd.3 ' entity='osd.3' 2024-03-21T13:09:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:23 smithi152 ceph-mon[34051]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-21T13:09:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:23 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:23.695 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T13:09:23.695 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T13:09:23.695 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-21T13:09:23.695 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-21T13:09:23.695 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113871 s, 921 MB/s 2024-03-21T13:09:23.695 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-21T13:09:24.327 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_4 2024-03-21T13:09:24.515 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:09:24.515 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:09:24.515 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[30095]: osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] boot 2024-03-21T13:09:24.515 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[30095]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T13:09:24.515 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:24.516 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:09:24.516 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:09:24.516 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[39209]: osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] boot 2024-03-21T13:09:24.516 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[39209]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T13:09:24.516 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:24 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:24.565 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:24 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:09:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:24 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:09:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:24 smithi152 ceph-mon[34051]: osd.3 [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] boot 2024-03-21T13:09:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:24 smithi152 ceph-mon[34051]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T13:09:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:09:25.656 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T13:09:25.656 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi089 2024-03-21T13:09:25.656 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.656 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.656 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.656 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.656 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi089 to 4016M 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:25 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T13:09:25.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi089 2024-03-21T13:09:25.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi089 to 4016M 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi089 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi089 to 4016M 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:25.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:25.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:25.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:25.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:25.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:25 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:26.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 -- 172.21.15.152:0/1939817029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41021f0 msgr2=0x7fe7c4102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:26.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1939817029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41021f0 0x7fe7c4102610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc00b0d0 tx=0x7fe7bc00b3e0 comp rx=0 tx=0).stop 2024-03-21T13:09:26.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 -- 172.21.15.152:0/1939817029 shutdown_connections 2024-03-21T13:09:26.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1939817029 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7c4103d90 0x7fe7c410a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1939817029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41021f0 0x7fe7c4102610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1939817029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c41033f0 0x7fe7c4103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 -- 172.21.15.152:0/1939817029 >> 172.21.15.152:0/1939817029 conn(0x7fe7c40fd790 msgr2=0x7fe7c40ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 -- 172.21.15.152:0/1939817029 shutdown_connections 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.173+0000 7fe7c894d700 1 -- 172.21.15.152:0/1939817029 wait complete. 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 Processor -- start 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 -- start start 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c41021f0 0x7fe7c41978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41033f0 0x7fe7c4197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7c4103d90 0x7fe7c419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c410d310 con 0x7fe7c41021f0 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c410d190 con 0x7fe7c41033f0 2024-03-21T13:09:26.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c894d700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c410d490 con 0x7fe7c4103d90 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41033f0 0x7fe7c4197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41033f0 0x7fe7c4197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:59328/0 (socket says 172.21.15.152:59328) 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c1d9b700 1 -- 172.21.15.152:0/1949313553 learned_addr learned my addr 172.21.15.152:0/1949313553 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c259c700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c41021f0 0x7fe7c41978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.174+0000 7fe7c2d9d700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7c4103d90 0x7fe7c419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c1d9b700 1 -- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7c4103d90 msgr2=0x7fe7c419eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c1d9b700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7c4103d90 0x7fe7c419eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c1d9b700 1 -- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c41021f0 msgr2=0x7fe7c41978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c1d9b700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c41021f0 0x7fe7c41978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c1d9b700 1 -- 172.21.15.152:0/1949313553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c419f550 con 0x7fe7c41033f0 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c259c700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c41021f0 0x7fe7c41978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:09:26.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c1d9b700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41033f0 0x7fe7c4197e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800c9b0 tx=0x7fe7b800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:09:26.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7b37fe700 1 -- 172.21.15.152:0/1949313553 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8007b00 con 0x7fe7c41033f0 2024-03-21T13:09:26.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7b37fe700 1 -- 172.21.15.152:0/1949313553 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe7b800ce50 con 0x7fe7c41033f0 2024-03-21T13:09:26.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c419f840 con 0x7fe7c41033f0 2024-03-21T13:09:26.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7b37fe700 1 -- 172.21.15.152:0/1949313553 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8018420 con 0x7fe7c41033f0 2024-03-21T13:09:26.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.175+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c419fd30 con 0x7fe7c41033f0 2024-03-21T13:09:26.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.176+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c4068e50 con 0x7fe7c41033f0 2024-03-21T13:09:26.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.177+0000 7fe7b37fe700 1 -- 172.21.15.152:0/1949313553 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7fe7b800f450 con 0x7fe7c41033f0 2024-03-21T13:09:26.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.177+0000 7fe7b37fe700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe7ac06e250 0x7fe7ac070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:26.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.177+0000 7fe7b37fe700 1 -- 172.21.15.152:0/1949313553 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3449+0+0 (secure 0 0 0) 0x7fe7b808d6b0 con 0x7fe7c41033f0 2024-03-21T13:09:26.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.178+0000 7fe7c259c700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe7ac06e250 0x7fe7ac070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:26.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.178+0000 7fe7c259c700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe7ac06e250 0x7fe7ac070710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc000f80 tx=0x7fe7bc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:09:26.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.182+0000 7fe7b37fe700 1 -- 172.21.15.152:0/1949313553 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe7b8014030 con 0x7fe7c41033f0 2024-03-21T13:09:26.390 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:26.388+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7c40625d0 con 0x7fe7ac06e250 2024-03-21T13:09:27.237 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:26 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:27.237 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:26 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:27.237 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:26 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:27.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:26 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:27.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:26 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:27.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:26 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:27.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:26 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:27.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:26 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:27.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:26 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:27.948 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:27 smithi152 ceph-mon[34051]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:27.949 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:27 smithi152 ceph-mon[34051]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:27 smithi089 ceph-mon[30095]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:27 smithi089 ceph-mon[30095]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:27 smithi089 ceph-mon[39209]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:27 smithi089 ceph-mon[39209]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:29 smithi152 ceph-mon[34051]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/1474125220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]: dispatch 2024-03-21T13:09:30.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]: dispatch 2024-03-21T13:09:30.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]': finished 2024-03-21T13:09:30.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:29 smithi152 ceph-mon[34051]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T13:09:30.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:29 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:30.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/1587269187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[30095]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/1474125220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]: dispatch 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]: dispatch 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]': finished 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[30095]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/1587269187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[39209]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/1474125220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]: dispatch 2024-03-21T13:09:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]: dispatch 2024-03-21T13:09:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d743da7-8b8c-40f8-a591-e791ed280233"}]': finished 2024-03-21T13:09:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[39209]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T13:09:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/1587269187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:32.283 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:31 smithi152 ceph-mon[34051]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:31 smithi089 ceph-mon[39209]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:31 smithi089 ceph-mon[30095]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:34.031 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:33 smithi152 ceph-mon[34051]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:33 smithi089 ceph-mon[39209]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:34.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:33 smithi089 ceph-mon[30095]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:36.029 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:35 smithi152 ceph-mon[34051]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:36.030 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T13:09:36.030 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:35 smithi089 ceph-mon[39209]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T13:09:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:35 smithi089 ceph-mon[30095]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T13:09:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:37.197 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:37 smithi152 ceph-mon[34051]: Deploying daemon osd.4 on smithi152 2024-03-21T13:09:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:37 smithi089 ceph-mon[39209]: Deploying daemon osd.4 on smithi152 2024-03-21T13:09:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:37 smithi089 ceph-mon[30095]: Deploying daemon osd.4 on smithi152 2024-03-21T13:09:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:38 smithi152 ceph-mon[34051]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:38.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:38 smithi089 ceph-mon[39209]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:38 smithi089 ceph-mon[30095]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:40.197 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:40 smithi152 ceph-mon[34051]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:40 smithi089 ceph-mon[30095]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:40 smithi089 ceph-mon[39209]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:41 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:42.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:41 smithi152 ceph-mon[34051]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[30095]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:42.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:41 smithi089 ceph-mon[39209]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:44.303 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:44.304 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.304 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.304 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:44.304 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:44.304 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.304 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: from='osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:09:44.304 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:43 smithi152 ceph-mon[34051]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:09:44.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.303+0000 7fe7b37fe700 1 -- 172.21.15.152:0/1949313553 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe7c40625d0 con 0x7fe7ac06e250 2024-03-21T13:09:44.305 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 4 on host 'smithi152' 2024-03-21T13:09:44.306 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe7ac06e250 msgr2=0x7fe7ac070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:44.306 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe7ac06e250 0x7fe7ac070710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc000f80 tx=0x7fe7bc00b040 comp rx=0 tx=0).stop 2024-03-21T13:09:44.306 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41033f0 msgr2=0x7fe7c4197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:44.306 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41033f0 0x7fe7c4197e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800c9b0 tx=0x7fe7b800ccc0 comp rx=0 tx=0).stop 2024-03-21T13:09:44.306 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 shutdown_connections 2024-03-21T13:09:44.306 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7c4103d90 0x7fe7c419eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:44.307 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c41021f0 0x7fe7c41978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:44.307 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe7ac06e250 0x7fe7ac070710 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:44.307 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 --2- 172.21.15.152:0/1949313553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7c41033f0 0x7fe7c4197e30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:44.307 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 >> 172.21.15.152:0/1949313553 conn(0x7fe7c40fd790 msgr2=0x7fe7c41066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:09:44.307 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 shutdown_connections 2024-03-21T13:09:44.307 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:44.305+0000 7fe7c894d700 1 -- 172.21.15.152:0/1949313553 wait complete. 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: from='osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:09:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[30095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: from='osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:09:44.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:43 smithi089 ceph-mon[39209]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:09:44.964 DEBUG:teuthology.orchestra.run.smithi152:osd.4> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.4.service 2024-03-21T13:09:44.966 INFO:tasks.cephadm:Deploying osd.5 on smithi152 with /dev/vg_nvme/lv_3... 2024-03-21T13:09:44.967 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T13:09:45.484 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:45.484 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:45.485 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:45.485 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T13:09:45.485 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: from='osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:09:45.485 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T13:09:45.485 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:45.485 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:09:45.485 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:45 smithi152 ceph-mon[34051]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:45.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:45.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:45.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:45.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T13:09:45.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: from='osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:09:45.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T13:09:45.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[30095]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: from='osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:09:45.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:45 smithi089 ceph-mon[39209]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:46.113 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:09:46.000+0000 7f3e88aa5700 -1 osd.4 0 waiting for initial osdmap 2024-03-21T13:09:46.114 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:09:46.007+0000 7f3e80890700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:09:46.702 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:46.810 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-mon[34051]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:09:46.810 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-mon[34051]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T13:09:46.810 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:46.810 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-mon[34051]: osdmap e31: 5 total, 4 up, 5 in 2024-03-21T13:09:46.810 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:46.811 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:46 smithi152 ceph-mon[34051]: from='osd.4 ' entity='osd.4' 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[30095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[30095]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[30095]: osdmap e31: 5 total, 4 up, 5 in 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[30095]: from='osd.4 ' entity='osd.4' 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[39209]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:09:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[39209]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T13:09:46.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:46.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[39209]: osdmap e31: 5 total, 4 up, 5 in 2024-03-21T13:09:46.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:46.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:46 smithi089 ceph-mon[39209]: from='osd.4 ' entity='osd.4' 2024-03-21T13:09:47.659 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:47 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:09:47.659 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:47 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:09:47.659 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:47.659 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:47 smithi152 ceph-mon[34051]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:47.659 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:47 smithi152 ceph-mon[34051]: osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] boot 2024-03-21T13:09:47.660 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:47 smithi152 ceph-mon[34051]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T13:09:47.660 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:09:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:09:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[39209]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[39209]: osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] boot 2024-03-21T13:09:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[39209]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T13:09:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:09:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:09:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[30095]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-21T13:09:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[30095]: osd.4 [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] boot 2024-03-21T13:09:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[30095]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T13:09:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:09:49.190 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T13:09:49.190 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T13:09:49.190 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-21T13:09:49.191 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-21T13:09:49.191 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125242 s, 837 MB/s 2024-03-21T13:09:49.191 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-21T13:09:49.213 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T13:09:49.213 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi152 2024-03-21T13:09:49.213 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.214 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.214 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:49.214 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi152 to 17088M 2024-03-21T13:09:49.214 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.214 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:49.214 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:49.214 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi152 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi152 to 17088M 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:49.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi152 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi152 to 17088M 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:09:49.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:09:49.748 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_3 2024-03-21T13:09:49.987 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:09:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:49 smithi152 ceph-mon[34051]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:09:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:49 smithi152 ceph-mon[34051]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T13:09:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:49 smithi089 ceph-mon[30095]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:09:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:49 smithi089 ceph-mon[30095]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T13:09:50.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:49 smithi089 ceph-mon[39209]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:09:50.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:49 smithi089 ceph-mon[39209]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T13:09:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:50 smithi152 ceph-mon[34051]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T13:09:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:50 smithi089 ceph-mon[30095]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T13:09:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:50 smithi089 ceph-mon[39209]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T13:09:51.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 -- 172.21.15.152:0/3533715471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe41010b0 msgr2=0x7effe41014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:51.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 --2- 172.21.15.152:0/3533715471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe41010b0 0x7effe41014d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7effd40099c0 tx=0x7effd4009cd0 comp rx=0 tx=0).stop 2024-03-21T13:09:51.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 -- 172.21.15.152:0/3533715471 shutdown_connections 2024-03-21T13:09:51.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 --2- 172.21.15.152:0/3533715471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe4102c50 0x7effe4109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:51.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 --2- 172.21.15.152:0/3533715471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe41010b0 0x7effe41014d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 --2- 172.21.15.152:0/3533715471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7effe41022b0 0x7effe4102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 -- 172.21.15.152:0/3533715471 >> 172.21.15.152:0/3533715471 conn(0x7effe40fc650 msgr2=0x7effe40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 -- 172.21.15.152:0/3533715471 shutdown_connections 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.542+0000 7effe9093700 1 -- 172.21.15.152:0/3533715471 wait complete. 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.543+0000 7effe9093700 1 Processor -- start 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.543+0000 7effe9093700 1 -- start start 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe9093700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe41010b0 0x7effe406b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe9093700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7effe41022b0 0x7effe419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe9093700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe4102c50 0x7effe419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe9093700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe410c1f0 con 0x7effe41010b0 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe9093700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe410c070 con 0x7effe4102c50 2024-03-21T13:09:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe9093700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe410c370 con 0x7effe41022b0 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe4102c50 0x7effe419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe4102c50 0x7effe419ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:33048/0 (socket says 172.21.15.152:33048) 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 -- 172.21.15.152:0/1193875748 learned_addr learned my addr 172.21.15.152:0/1193875748 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe2d9d700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe41010b0 0x7effe406b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe259c700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7effe41022b0 0x7effe419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 -- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7effe41022b0 msgr2=0x7effe419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7effe41022b0 0x7effe419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 -- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe41010b0 msgr2=0x7effe406b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe41010b0 0x7effe406b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 -- 172.21.15.152:0/1193875748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe419f600 con 0x7effe4102c50 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe359e700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe4102c50 0x7effe419ed30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7effdc007ee0 tx=0x7effdc00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effcbfff700 1 -- 172.21.15.152:0/1193875748 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc004bb0 con 0x7effe4102c50 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe419f8f0 con 0x7effe4102c50 2024-03-21T13:09:51.547 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effcbfff700 1 -- 172.21.15.152:0/1193875748 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7effdc004d50 con 0x7effe4102c50 2024-03-21T13:09:51.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.544+0000 7effcbfff700 1 -- 172.21.15.152:0/1193875748 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc0057a0 con 0x7effe4102c50 2024-03-21T13:09:51.548 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.545+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe419fe30 con 0x7effe4102c50 2024-03-21T13:09:51.549 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.545+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe4068e50 con 0x7effe4102c50 2024-03-21T13:09:51.549 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.546+0000 7effcbfff700 1 -- 172.21.15.152:0/1193875748 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7effdc01a020 con 0x7effe4102c50 2024-03-21T13:09:51.549 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.546+0000 7effcbfff700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7effcc06e250 0x7effcc070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:09:51.549 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.546+0000 7effcbfff700 1 -- 172.21.15.152:0/1193875748 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 3991+0+0 (secure 0 0 0) 0x7effdc08d060 con 0x7effe4102c50 2024-03-21T13:09:51.549 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.547+0000 7effe2d9d700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7effcc06e250 0x7effcc070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:09:51.549 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.547+0000 7effe2d9d700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7effcc06e250 0x7effcc070710 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7effd400b580 tx=0x7effd4005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:09:51.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.551+0000 7effcbfff700 1 -- 172.21.15.152:0/1193875748 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7effdc010070 con 0x7effe4102c50 2024-03-21T13:09:51.753 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:09:51.752+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7effe402de40 con 0x7effcc06e250 2024-03-21T13:09:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:51 smithi152 ceph-mon[34051]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-21T13:09:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:51 smithi152 ceph-mon[34051]: osdmap e36: 5 total, 5 up, 5 in 2024-03-21T13:09:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:51 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:51 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:51 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:52.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[30095]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[30095]: osdmap e36: 5 total, 5 up, 5 in 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[39209]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[39209]: osdmap e36: 5 total, 5 up, 5 in 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:09:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:51 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:09:53.312 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:52 smithi152 ceph-mon[34051]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:52 smithi089 ceph-mon[30095]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:53.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:52 smithi089 ceph-mon[39209]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:09:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:53 smithi152 ceph-mon[34051]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-21T13:09:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:53 smithi089 ceph-mon[30095]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-21T13:09:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:53 smithi089 ceph-mon[39209]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-21T13:09:55.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/2245156673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]: dispatch 2024-03-21T13:09:55.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]: dispatch 2024-03-21T13:09:55.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]': finished 2024-03-21T13:09:55.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:54 smithi152 ceph-mon[34051]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T13:09:55.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:54 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/2245156673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]: dispatch 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]: dispatch 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]': finished 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[30095]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/2245156673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]: dispatch 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]: dispatch 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2ece7d8-4a4a-4247-b3af-46c854bcc1dc"}]': finished 2024-03-21T13:09:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[39209]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T13:09:55.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:54 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:09:56.235 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:55 smithi152 ceph-mon[34051]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T13:09:56.235 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/3874480627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:56.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:55 smithi089 ceph-mon[30095]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T13:09:56.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/3874480627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:56.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:55 smithi089 ceph-mon[39209]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T13:09:56.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/3874480627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:09:58.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:57 smithi152 ceph-mon[34051]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-21T13:09:58.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:57 smithi089 ceph-mon[30095]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-21T13:09:58.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:57 smithi089 ceph-mon[39209]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-21T13:10:00.236 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:09:59 smithi152 ceph-mon[34051]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:09:59 smithi089 ceph-mon[30095]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:09:59 smithi089 ceph-mon[39209]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:01.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:00 smithi152 ceph-mon[34051]: overall HEALTH_OK 2024-03-21T13:10:01.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:00 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T13:10:01.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:00 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:01.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:00 smithi089 ceph-mon[30095]: overall HEALTH_OK 2024-03-21T13:10:01.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:00 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T13:10:01.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:00 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:01.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:00 smithi089 ceph-mon[39209]: overall HEALTH_OK 2024-03-21T13:10:01.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:00 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T13:10:01.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:00 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:01 smithi152 ceph-mon[34051]: Deploying daemon osd.5 on smithi152 2024-03-21T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:01 smithi152 ceph-mon[34051]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:01 smithi089 ceph-mon[30095]: Deploying daemon osd.5 on smithi152 2024-03-21T13:10:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:01 smithi089 ceph-mon[30095]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:01 smithi089 ceph-mon[39209]: Deploying daemon osd.5 on smithi152 2024-03-21T13:10:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:01 smithi089 ceph-mon[39209]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:03 smithi152 ceph-mon[34051]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:04.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:03 smithi089 ceph-mon[30095]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:03 smithi089 ceph-mon[39209]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:05 smithi152 ceph-mon[34051]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:06.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:05 smithi089 ceph-mon[30095]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:05 smithi089 ceph-mon[39209]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:07 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:07.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:07 smithi152 ceph-mon[34051]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:07.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:07.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[30095]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:07.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:07.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:07.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:07.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:07 smithi089 ceph-mon[39209]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:09.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.761+0000 7effcbfff700 1 -- 172.21.15.152:0/1193875748 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7effe402de40 con 0x7effcc06e250 2024-03-21T13:10:09.762 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 5 on host 'smithi152' 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7effcc06e250 msgr2=0x7effcc070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7effcc06e250 0x7effcc070710 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7effd400b580 tx=0x7effd4005bc0 comp rx=0 tx=0).stop 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe4102c50 msgr2=0x7effe419ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe4102c50 0x7effe419ed30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7effdc007ee0 tx=0x7effdc00d3b0 comp rx=0 tx=0).stop 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 shutdown_connections 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7effcc06e250 0x7effcc070710 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7effe4102c50 0x7effe419ed30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe41010b0 0x7effe406b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 --2- 172.21.15.152:0/1193875748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7effe41022b0 0x7effe419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 >> 172.21.15.152:0/1193875748 conn(0x7effe40fc650 msgr2=0x7effe41056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 shutdown_connections 2024-03-21T13:10:09.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:09.764+0000 7effe9093700 1 -- 172.21.15.152:0/1193875748 wait complete. 2024-03-21T13:10:09.998 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[30095]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:09.998 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[30095]: from='osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:10:09.998 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[30095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:10:09.998 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:09.998 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:09.998 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[39209]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:09.999 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[39209]: from='osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:10:09.999 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[39209]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:10:09.999 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:09.999 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:09 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:10.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:09 smithi152 ceph-mon[34051]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:10.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:09 smithi152 ceph-mon[34051]: from='osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:10:10.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:09 smithi152 ceph-mon[34051]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:10:10.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:09 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:10.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:09 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:10.289 DEBUG:teuthology.orchestra.run.smithi152:osd.5> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.5.service 2024-03-21T13:10:10.291 INFO:tasks.cephadm:Deploying osd.6 on smithi152 with /dev/vg_nvme/lv_2... 2024-03-21T13:10:10.291 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T13:10:10.577 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:10:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-mon[34051]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T13:10:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-mon[34051]: from='osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-mon[34051]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T13:10:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-mon[34051]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[30095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[30095]: from='osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[30095]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[30095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[39209]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T13:10:11.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[39209]: from='osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[39209]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T13:10:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[39209]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:11 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:11.665 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:10:11.431+0000 7f9dda484700 -1 osd.5 0 waiting for initial osdmap 2024-03-21T13:10:11.666 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:10:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:10:11.436+0000 7f9dd1a6e700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:10:12.233 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:12.233 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:10:12.234 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: osdmap e39: 6 total, 5 up, 6 in 2024-03-21T13:10:12.234 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.234 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.234 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: from='osd.5 ' entity='osd.5' 2024-03-21T13:10:12.234 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] boot 2024-03-21T13:10:12.234 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T13:10:12.234 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:10:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: osdmap e39: 6 total, 5 up, 6 in 2024-03-21T13:10:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: from='osd.5 ' entity='osd.5' 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] boot 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: osdmap e39: 6 total, 5 up, 6 in 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: from='osd.5 ' entity='osd.5' 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: osd.5 [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] boot 2024-03-21T13:10:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T13:10:12.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:10:12.476 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T13:10:12.477 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T13:10:12.477 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-21T13:10:12.477 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-21T13:10:12.477 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014183 s, 739 MB/s 2024-03-21T13:10:12.477 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-21T13:10:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:13 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:10:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:13 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:10:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:13 smithi152 ceph-mon[34051]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T13:10:13.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_2 2024-03-21T13:10:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:13 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:10:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:13 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:10:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:13 smithi089 ceph-mon[30095]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T13:10:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:13 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:10:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:13 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:10:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:13 smithi089 ceph-mon[39209]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T13:10:13.742 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:10:14.306 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:14 smithi152 ceph-mon[34051]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:14.306 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:14 smithi152 ceph-mon[34051]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T13:10:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:14 smithi089 ceph-mon[30095]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:14 smithi089 ceph-mon[30095]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T13:10:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:14 smithi089 ceph-mon[39209]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:14 smithi089 ceph-mon[39209]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:16.138 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:16.139 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.139 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:16.139 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:16.139 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:16.139 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:16.353 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 -- 172.21.15.152:0/1502318843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c103d60 msgr2=0x7fe54c10a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:16.354 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1502318843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c103d60 0x7fe54c10a610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe54400b0d0 tx=0x7fe54400b3e0 comp rx=0 tx=0).stop 2024-03-21T13:10:16.354 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 -- 172.21.15.152:0/1502318843 shutdown_connections 2024-03-21T13:10:16.354 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1502318843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c103d60 0x7fe54c10a610 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:16.354 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1502318843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54c1021c0 0x7fe54c1025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:16.354 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1502318843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54c1033c0 0x7fe54c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:16.354 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 -- 172.21.15.152:0/1502318843 >> 172.21.15.152:0/1502318843 conn(0x7fe54c0fd7a0 msgr2=0x7fe54c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 -- 172.21.15.152:0/1502318843 shutdown_connections 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 -- 172.21.15.152:0/1502318843 wait complete. 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 Processor -- start 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.350+0000 7fe551e7f700 1 -- start start 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe551e7f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c1021c0 0x7fe54c197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe551e7f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54c1033c0 0x7fe54c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe551e7f700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54c103d60 0x7fe54c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe551e7f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c10d2b0 con 0x7fe54c1033c0 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe551e7f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c10d130 con 0x7fe54c1021c0 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe551e7f700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c10d430 con 0x7fe54c103d60 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c1021c0 0x7fe54c197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c1021c0 0x7fe54c197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:46714/0 (socket says 172.21.15.152:46714) 2024-03-21T13:10:16.355 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 -- 172.21.15.152:0/1870571231 learned_addr learned my addr 172.21.15.152:0/1870571231 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe542ffd700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54c1033c0 0x7fe54c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54bfff700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54c103d60 0x7fe54c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 -- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54c103d60 msgr2=0x7fe54c19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54c103d60 0x7fe54c19ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 -- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54c1033c0 msgr2=0x7fe54c197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54c1033c0 0x7fe54c197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 -- 172.21.15.152:0/1870571231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c19f3b0 con 0x7fe54c1021c0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54bfff700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54c103d60 0x7fe54c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe54b7fe700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c1021c0 0x7fe54c197750 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00c990 tx=0x7fe53c00cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.351+0000 7fe5497fa700 1 -- 172.21.15.152:0/1870571231 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c007b00 con 0x7fe54c1021c0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.352+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19f6a0 con 0x7fe54c1021c0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.352+0000 7fe5497fa700 1 -- 172.21.15.152:0/1870571231 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe53c007ca0 con 0x7fe54c1021c0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.352+0000 7fe5497fa700 1 -- 172.21.15.152:0/1870571231 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c018420 con 0x7fe54c1021c0 2024-03-21T13:10:16.356 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.352+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c19fc30 con 0x7fe54c1021c0 2024-03-21T13:10:16.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.353+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c068e50 con 0x7fe54c1021c0 2024-03-21T13:10:16.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.353+0000 7fe5497fa700 1 -- 172.21.15.152:0/1870571231 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7fe53c00f450 con 0x7fe54c1021c0 2024-03-21T13:10:16.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.354+0000 7fe5497fa700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe53806e250 0x7fe538070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:16.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.354+0000 7fe5497fa700 1 -- 172.21.15.152:0/1870571231 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4423+0+0 (secure 0 0 0) 0x7fe53c08da80 con 0x7fe54c1021c0 2024-03-21T13:10:16.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.354+0000 7fe542ffd700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe53806e250 0x7fe538070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:16.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.354+0000 7fe542ffd700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe53806e250 0x7fe538070710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe53400ac50 tx=0x7fe53400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:10:16.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.358+0000 7fe5497fa700 1 -- 172.21.15.152:0/1870571231 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe53c014030 con 0x7fe54c1021c0 2024-03-21T13:10:16.560 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:16.560+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe54c0625d0 con 0x7fe53806e250 2024-03-21T13:10:17.275 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:17 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi152 2024-03-21T13:10:17.275 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:17 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi152 to 8544M 2024-03-21T13:10:17.275 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:17 smithi152 ceph-mon[34051]: osdmap e44: 6 total, 6 up, 6 in 2024-03-21T13:10:17.275 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:10:17.275 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:10:17.275 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:17.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi152 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi152 to 8544M 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[30095]: osdmap e44: 6 total, 6 up, 6 in 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi152 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi152 to 8544M 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[39209]: osdmap e44: 6 total, 6 up, 6 in 2024-03-21T13:10:17.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:10:17.330 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:10:17.330 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:18.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:18 smithi152 ceph-mon[34051]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:10:18.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:18 smithi152 ceph-mon[34051]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:18 smithi089 ceph-mon[30095]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:10:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:18 smithi089 ceph-mon[30095]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:18 smithi089 ceph-mon[39209]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:10:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:18 smithi089 ceph-mon[39209]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:20 smithi152 ceph-mon[34051]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/34416224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]: dispatch 2024-03-21T13:10:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]: dispatch 2024-03-21T13:10:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]': finished 2024-03-21T13:10:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:20 smithi152 ceph-mon[34051]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T13:10:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:20 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/93071274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[30095]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/34416224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]: dispatch 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]: dispatch 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]': finished 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[30095]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/93071274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[39209]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/34416224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]: dispatch 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]: dispatch 2024-03-21T13:10:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "877ec6ea-e165-4d53-aa88-87004c5b26c9"}]': finished 2024-03-21T13:10:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[39209]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T13:10:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/93071274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:10:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:22 smithi152 ceph-mon[34051]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:22 smithi089 ceph-mon[30095]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:22 smithi089 ceph-mon[39209]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:24.294 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:24 smithi152 ceph-mon[34051]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:24 smithi089 ceph-mon[30095]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:24 smithi089 ceph-mon[39209]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:26.296 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:26 smithi152 ceph-mon[34051]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:26.296 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:26 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T13:10:26.296 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:26 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:26 smithi089 ceph-mon[30095]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:26 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T13:10:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:26 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:26.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:26 smithi089 ceph-mon[39209]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:26.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:26 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T13:10:26.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:26 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:27 smithi152 ceph-mon[34051]: Deploying daemon osd.6 on smithi152 2024-03-21T13:10:27.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:27 smithi089 ceph-mon[30095]: Deploying daemon osd.6 on smithi152 2024-03-21T13:10:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:27 smithi089 ceph-mon[39209]: Deploying daemon osd.6 on smithi152 2024-03-21T13:10:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:28 smithi152 ceph-mon[34051]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:28 smithi089 ceph-mon[30095]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:28 smithi089 ceph-mon[39209]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:29 smithi152 ceph-mon[34051]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:29.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:29 smithi089 ceph-mon[30095]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:29 smithi089 ceph-mon[39209]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:32.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:32 smithi152 ceph-mon[34051]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:32.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:32.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:32.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:32 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[30095]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[39209]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:32.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:32 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:34 smithi152 ceph-mon[34051]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:34.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:34 smithi089 ceph-mon[30095]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:34 smithi089 ceph-mon[39209]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:35.075 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 6 on host 'smithi152' 2024-03-21T13:10:35.075 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.073+0000 7fe5497fa700 1 -- 172.21.15.152:0/1870571231 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe54c0625d0 con 0x7fe53806e250 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe53806e250 msgr2=0x7fe538070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe53806e250 0x7fe538070710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe53400ac50 tx=0x7fe53400a380 comp rx=0 tx=0).stop 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c1021c0 msgr2=0x7fe54c197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c1021c0 0x7fe54c197750 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00c990 tx=0x7fe53c00cca0 comp rx=0 tx=0).stop 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 shutdown_connections 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54c103d60 0x7fe54c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fe53806e250 0x7fe538070710 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54c1021c0 0x7fe54c197750 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 --2- 172.21.15.152:0/1870571231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54c1033c0 0x7fe54c197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 >> 172.21.15.152:0/1870571231 conn(0x7fe54c0fd7a0 msgr2=0x7fe54c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:10:35.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 shutdown_connections 2024-03-21T13:10:35.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:35.077+0000 7fe551e7f700 1 -- 172.21.15.152:0/1870571231 wait complete. 2024-03-21T13:10:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:35 smithi152 ceph-mon[34051]: from='osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:10:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:35 smithi152 ceph-mon[34051]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:10:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:35 smithi152 ceph-mon[34051]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[39209]: from='osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:10:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[39209]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:10:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[39209]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[30095]: from='osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:10:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[30095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:10:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[30095]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:35.708 DEBUG:teuthology.orchestra.run.smithi152:osd.6> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.6.service 2024-03-21T13:10:35.711 INFO:tasks.cephadm:Deploying osd.7 on smithi152 with /dev/vg_nvme/lv_1... 2024-03-21T13:10:35.711 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T13:10:36.468 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:10:36.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:36 smithi152 ceph-mon[34051]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T13:10:36.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:36 smithi152 ceph-mon[34051]: from='osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:36.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:36 smithi152 ceph-mon[34051]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T13:10:36.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:36 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:36.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:36 smithi152 ceph-mon[34051]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[39209]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T13:10:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[39209]: from='osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[39209]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T13:10:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[39209]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[30095]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T13:10:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[30095]: from='osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[30095]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T13:10:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:36 smithi089 ceph-mon[30095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:10:36.821 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:10:36 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:10:36.692+0000 7f2b1f4e0700 -1 osd.6 0 waiting for initial osdmap 2024-03-21T13:10:36.821 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:10:36 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:10:36.702+0000 7f2b172cb700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: osdmap e47: 7 total, 6 up, 7 in 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: from='osd.6 ' entity='osd.6' 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] boot 2024-03-21T13:10:37.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:37 smithi152 ceph-mon[34051]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T13:10:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:10:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: osdmap e47: 7 total, 6 up, 7 in 2024-03-21T13:10:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: from='osd.6 ' entity='osd.6' 2024-03-21T13:10:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] boot 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[39209]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: osdmap e47: 7 total, 6 up, 7 in 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: from='osd.6 ' entity='osd.6' 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-21T13:10:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: osd.6 [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] boot 2024-03-21T13:10:37.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:37 smithi089 ceph-mon[30095]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T13:10:38.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:38 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:10:38.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:38 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:10:38.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:38 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:38.392 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:38 smithi152 ceph-mon[34051]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T13:10:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:10:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:10:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[30095]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T13:10:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:10:38.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:10:38.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:10:38.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:38 smithi089 ceph-mon[39209]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T13:10:39.348 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T13:10:39.348 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T13:10:39.348 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-21T13:10:39.348 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-21T13:10:39.348 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122626 s, 855 MB/s 2024-03-21T13:10:39.348 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-21T13:10:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:39 smithi152 ceph-mon[34051]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 593 MiB used, 625 GiB / 626 GiB avail 2024-03-21T13:10:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:39 smithi152 ceph-mon[34051]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T13:10:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:39 smithi089 ceph-mon[30095]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 593 MiB used, 625 GiB / 626 GiB avail 2024-03-21T13:10:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:39 smithi089 ceph-mon[30095]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T13:10:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:39 smithi089 ceph-mon[39209]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 593 MiB used, 625 GiB / 626 GiB avail 2024-03-21T13:10:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:39 smithi089 ceph-mon[39209]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T13:10:39.965 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_1 2024-03-21T13:10:40.204 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:10:40.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi152 2024-03-21T13:10:40.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi152 to 5696M 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:40 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi152 2024-03-21T13:10:40.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi152 to 5696M 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:40.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi152 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi152 to 5696M 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:40.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:40.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:10:40.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:40 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 -- 172.21.15.152:0/4061495919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 msgr2=0x7f34e8109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 --2- 172.21.15.152:0/4061495919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e8109530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f34e000b0d0 tx=0x7f34e000b3e0 comp rx=0 tx=0).stop 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 -- 172.21.15.152:0/4061495919 shutdown_connections 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 --2- 172.21.15.152:0/4061495919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e8109530 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 --2- 172.21.15.152:0/4061495919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34e81010e0 0x7f34e8101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 --2- 172.21.15.152:0/4061495919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34e81022e0 0x7f34e8102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 -- 172.21.15.152:0/4061495919 >> 172.21.15.152:0/4061495919 conn(0x7f34e80fc680 msgr2=0x7f34e80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 -- 172.21.15.152:0/4061495919 shutdown_connections 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.730+0000 7f34eedc2700 1 -- 172.21.15.152:0/4061495919 wait complete. 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 Processor -- start 2024-03-21T13:10:41.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 -- start start 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34e81010e0 0x7f34e806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34e81022e0 0x7f34e819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e810c210 con 0x7f34e81010e0 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e810c090 con 0x7f34e8102c80 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34eedc2700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e810c390 con 0x7f34e81022e0 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34ed35f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34ed35f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e819ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:32794/0 (socket says 172.21.15.152:32794) 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.731+0000 7f34ed35f700 1 -- 172.21.15.152:0/1914123192 learned_addr learned my addr 172.21.15.152:0/1914123192 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34ecb5e700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34e81010e0 0x7f34e806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34e7fff700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34e81022e0 0x7f34e819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34ed35f700 1 -- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34e81022e0 msgr2=0x7f34e819c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:41.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34ed35f700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34e81022e0 0x7f34e819c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34ed35f700 1 -- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34e81010e0 msgr2=0x7f34e806b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34ed35f700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34e81010e0 0x7f34e806b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34ed35f700 1 -- 172.21.15.152:0/1914123192 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e819f510 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34ed35f700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e819ec40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34e0007d90 tx=0x7f34e0009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34e5ffb700 1 -- 172.21.15.152:0/1914123192 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0022070 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34e5ffb700 1 -- 172.21.15.152:0/1914123192 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f34e0013d90 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e819f7a0 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34e5ffb700 1 -- 172.21.15.152:0/1914123192 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e001bcd0 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.732+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e819fc50 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.733+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e8068e50 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.734+0000 7f34e5ffb700 1 -- 172.21.15.152:0/1914123192 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7f34e0025070 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.734+0000 7f34e5ffb700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f34d806e250 0x7f34d8070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.734+0000 7f34e5ffb700 1 -- 172.21.15.152:0/1914123192 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4855+0+0 (secure 0 0 0) 0x7f34e000f020 con 0x7f34e8102c80 2024-03-21T13:10:41.739 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.734+0000 7f34ecb5e700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f34d806e250 0x7f34d8070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:10:41.740 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.735+0000 7f34ecb5e700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f34d806e250 0x7f34d8070710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f34dc005950 tx=0x7f34dc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:10:41.740 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.739+0000 7f34e5ffb700 1 -- 172.21.15.152:0/1914123192 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f34e002c020 con 0x7f34e8102c80 2024-03-21T13:10:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:41 smithi152 ceph-mon[34051]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T13:10:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:41 smithi152 ceph-mon[34051]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 597 MiB used, 625 GiB / 626 GiB avail 2024-03-21T13:10:41.944 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:10:41.943+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f34e802dd20 con 0x7f34d806e250 2024-03-21T13:10:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:41 smithi089 ceph-mon[39209]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T13:10:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:41 smithi089 ceph-mon[39209]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 597 MiB used, 625 GiB / 626 GiB avail 2024-03-21T13:10:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:41 smithi089 ceph-mon[30095]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T13:10:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:41 smithi089 ceph-mon[30095]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 597 MiB used, 625 GiB / 626 GiB avail 2024-03-21T13:10:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:42 smithi152 ceph-mon[34051]: osdmap e52: 7 total, 7 up, 7 in 2024-03-21T13:10:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:42 smithi152 ceph-mon[34051]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:10:42.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:42 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:10:42.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:42 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:10:42.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:42 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[30095]: osdmap e52: 7 total, 7 up, 7 in 2024-03-21T13:10:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[30095]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:10:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:10:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:10:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[39209]: osdmap e52: 7 total, 7 up, 7 in 2024-03-21T13:10:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[39209]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:10:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:10:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:10:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:42 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:43 smithi152 ceph-mon[34051]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-21T13:10:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:43 smithi089 ceph-mon[30095]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-21T13:10:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:43 smithi089 ceph-mon[39209]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-21T13:10:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/264782943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]: dispatch 2024-03-21T13:10:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]: dispatch 2024-03-21T13:10:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]': finished 2024-03-21T13:10:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:44 smithi152 ceph-mon[34051]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T13:10:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:44 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:10:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/264782943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]: dispatch 2024-03-21T13:10:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]: dispatch 2024-03-21T13:10:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]': finished 2024-03-21T13:10:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[30095]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T13:10:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:10:45.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/264782943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]: dispatch 2024-03-21T13:10:45.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]: dispatch 2024-03-21T13:10:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "787eb11f-c37b-4862-b1e7-485a9da2851e"}]': finished 2024-03-21T13:10:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[39209]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T13:10:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:44 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:10:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:45 smithi152 ceph-mon[34051]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-21T13:10:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/2907774340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:10:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:45 smithi089 ceph-mon[30095]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-21T13:10:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/2907774340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:10:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:45 smithi089 ceph-mon[39209]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-21T13:10:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/2907774340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:10:48.241 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:48 smithi152 ceph-mon[34051]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-21T13:10:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:48 smithi089 ceph-mon[30095]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-21T13:10:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:48 smithi089 ceph-mon[39209]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-21T13:10:50.177 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:50 smithi152 ceph-mon[34051]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T13:10:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:50 smithi089 ceph-mon[30095]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T13:10:50.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:50 smithi089 ceph-mon[39209]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T13:10:52.110 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:52 smithi152 ceph-mon[34051]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-21T13:10:52.110 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:52 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T13:10:52.110 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:52 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:52 smithi089 ceph-mon[30095]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-21T13:10:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:52 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T13:10:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:52 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:52 smithi089 ceph-mon[39209]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-21T13:10:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:52 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T13:10:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:52 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:10:53.190 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:53 smithi152 ceph-mon[34051]: Deploying daemon osd.7 on smithi152 2024-03-21T13:10:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:53 smithi089 ceph-mon[30095]: Deploying daemon osd.7 on smithi152 2024-03-21T13:10:53.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:53 smithi089 ceph-mon[39209]: Deploying daemon osd.7 on smithi152 2024-03-21T13:10:54.287 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:54 smithi152 ceph-mon[34051]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:54 smithi089 ceph-mon[30095]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:54 smithi089 ceph-mon[39209]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:56.147 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:56 smithi152 ceph-mon[34051]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:56 smithi089 ceph-mon[30095]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:56 smithi089 ceph-mon[39209]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:57.780 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:57 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:57.780 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:57 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:57.780 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:57 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:57.780 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:10:57 smithi152 ceph-mon[34051]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:58.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:58.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:58.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:58.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[30095]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:10:58.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:58.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:10:58.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:10:58.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:10:57 smithi089 ceph-mon[39209]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:11:00.186 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:11:00.186 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.186 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.186 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:00.186 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:00.186 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.186 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: from='osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:11:00.187 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:00 smithi152 ceph-mon[34051]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:11:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:11:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: from='osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[30095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: from='osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:11:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:00 smithi089 ceph-mon[39209]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:11:00.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.709+0000 7f34e5ffb700 1 -- 172.21.15.152:0/1914123192 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f34e802dd20 con 0x7f34d806e250 2024-03-21T13:11:00.710 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 7 on host 'smithi152' 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f34d806e250 msgr2=0x7f34d8070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f34d806e250 0x7f34d8070710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f34dc005950 tx=0x7f34dc0058e0 comp rx=0 tx=0).stop 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 msgr2=0x7f34e819ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e819ec40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34e0007d90 tx=0x7f34e0009c30 comp rx=0 tx=0).stop 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 shutdown_connections 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f34d806e250 0x7f34d8070710 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34e8102c80 0x7f34e819ec40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:00.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34e81010e0 0x7f34e806b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:00.713 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 --2- 172.21.15.152:0/1914123192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34e81022e0 0x7f34e819c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:00.713 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 >> 172.21.15.152:0/1914123192 conn(0x7f34e80fc680 msgr2=0x7f34e81055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:00.713 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 shutdown_connections 2024-03-21T13:11:00.713 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:00.711+0000 7f34eedc2700 1 -- 172.21.15.152:0/1914123192 wait complete. 2024-03-21T13:11:01.361 DEBUG:teuthology.orchestra.run.smithi152:osd.7> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.7.service 2024-03-21T13:11:01.364 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-21T13:11:01.364 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T13:11:01.613 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:01.683 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: from='osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:01.684 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-mon[34051]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: from='osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:01.735 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[30095]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: from='osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:01.736 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:01 smithi089 ceph-mon[39209]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-21T13:11:01.954 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:11:01.836+0000 7fbd8ff62700 -1 osd.7 0 waiting for initial osdmap 2024-03-21T13:11:01.954 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:11:01 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:11:01.843+0000 7fbd87d4d700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:11:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:11:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T13:11:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: from='osd.7 ' entity='osd.7' 2024-03-21T13:11:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] boot 2024-03-21T13:11:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T13:11:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:02 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.676 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:11:02.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T13:11:02.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: from='osd.7 ' entity='osd.7' 2024-03-21T13:11:02.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] boot 2024-03-21T13:11:02.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T13:11:02.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: from='osd.7 ' entity='osd.7' 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: osd.7 [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] boot 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T13:11:02.678 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:02 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 -- 172.21.15.89:0/157863539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641033e0 msgr2=0x7fae64103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 --2- 172.21.15.89:0/157863539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641033e0 0x7fae64103840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fae540099c0 tx=0x7fae54009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 -- 172.21.15.89:0/157863539 shutdown_connections 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 --2- 172.21.15.89:0/157863539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae64103d80 0x7fae6410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 --2- 172.21.15.89:0/157863539 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae641021e0 0x7fae64102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 --2- 172.21.15.89:0/157863539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641033e0 0x7fae64103840 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 -- 172.21.15.89:0/157863539 >> 172.21.15.89:0/157863539 conn(0x7fae640fd7a0 msgr2=0x7fae640ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.163+0000 7fae6c71b700 1 -- 172.21.15.89:0/157863539 shutdown_connections 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 -- 172.21.15.89:0/157863539 wait complete. 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 Processor -- start 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 -- start start 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641021e0 0x7fae64197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae641033e0 0x7fae64197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae64103d80 0x7fae6419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410d2c0 con 0x7fae641021e0 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410d140 con 0x7fae641033e0 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.164+0000 7fae6c71b700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410d440 con 0x7fae64103d80 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641021e0 0x7fae64197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6acb8700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae64103d80 0x7fae6419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641021e0 0x7fae64197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42320/0 (socket says 172.21.15.89:42320) 2024-03-21T13:11:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 -- 172.21.15.89:0/4074425125 learned_addr learned my addr 172.21.15.89:0/4074425125 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae69cb6700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae641033e0 0x7fae64197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 -- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae64103d80 msgr2=0x7fae6419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae64103d80 0x7fae6419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 -- 172.21.15.89:0/4074425125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae641033e0 msgr2=0x7fae64197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae641033e0 0x7fae64197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 -- 172.21.15.89:0/4074425125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6419f560 con 0x7fae641021e0 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6acb8700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae64103d80 0x7fae6419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6a4b7700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641021e0 0x7fae64197900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae6000ea80 tx=0x7fae6000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae5b7fe700 1 -- 172.21.15.89:0/4074425125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6000cbd0 con 0x7fae641021e0 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae5b7fe700 1 -- 172.21.15.89:0/4074425125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fae6000cd70 con 0x7fae641021e0 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6419f850 con 0x7fae641021e0 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae5b7fe700 1 -- 172.21.15.89:0/4074425125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60018680 con 0x7fae641021e0 2024-03-21T13:11:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.165+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6419fd90 con 0x7fae641021e0 2024-03-21T13:11:03.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.166+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae64068e50 con 0x7fae641021e0 2024-03-21T13:11:03.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.167+0000 7fae5b7fe700 1 -- 172.21.15.89:0/4074425125 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7fae6001f020 con 0x7fae641021e0 2024-03-21T13:11:03.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.167+0000 7fae5b7fe700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fae5006e250 0x7fae50070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:03.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.167+0000 7fae5b7fe700 1 -- 172.21.15.89:0/4074425125 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5320+0+0 (secure 0 0 0) 0x7fae6008de20 con 0x7fae641021e0 2024-03-21T13:11:03.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.167+0000 7fae69cb6700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fae5006e250 0x7fae50070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:03.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.168+0000 7fae69cb6700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fae5006e250 0x7fae50070710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae54009750 tx=0x7fae54005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:03.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.172+0000 7fae5b7fe700 1 -- 172.21.15.89:0/4074425125 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae6005b920 con 0x7fae641021e0 2024-03-21T13:11:03.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.366+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fae6402ddc0 con 0x7fae641021e0 2024-03-21T13:11:03.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.367+0000 7fae5b7fe700 1 -- 172.21.15.89:0/4074425125 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v56) v1 ==== 74+0+130 (secure 0 0 0) 0x7fae6005b220 con 0x7fae641021e0 2024-03-21T13:11:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fae5006e250 msgr2=0x7fae50070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fae5006e250 0x7fae50070710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae54009750 tx=0x7fae54005fb0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641021e0 msgr2=0x7fae64197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641021e0 0x7fae64197900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae6000ea80 tx=0x7fae6000ed90 comp rx=0 tx=0).stop 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 shutdown_connections 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae64103d80 0x7fae6419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae641021e0 0x7fae64197900 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fae5006e250 0x7fae50070710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 --2- 172.21.15.89:0/4074425125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae641033e0 0x7fae64197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:03.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 >> 172.21.15.89:0/4074425125 conn(0x7fae640fd7a0 msgr2=0x7fae64106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:03.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 shutdown_connections 2024-03-21T13:11:03.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:03.369+0000 7fae6c71b700 1 -- 172.21.15.89:0/4074425125 wait complete. 2024-03-21T13:11:03.668 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:03 smithi152 ceph-mon[34051]: purged_snaps scrub starts 2024-03-21T13:11:03.668 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:03 smithi152 ceph-mon[34051]: purged_snaps scrub ok 2024-03-21T13:11:03.668 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:03 smithi152 ceph-mon[34051]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:03.668 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4074425125' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T13:11:03.668 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:03 smithi152 ceph-mon[34051]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T13:11:03.750 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[39209]: purged_snaps scrub starts 2024-03-21T13:11:03.750 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[39209]: purged_snaps scrub ok 2024-03-21T13:11:03.750 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[39209]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:03.751 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4074425125' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T13:11:03.751 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[39209]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T13:11:03.751 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[30095]: purged_snaps scrub starts 2024-03-21T13:11:03.751 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[30095]: purged_snaps scrub ok 2024-03-21T13:11:03.751 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[30095]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:03.751 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4074425125' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T13:11:03.751 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:03 smithi089 ceph-mon[30095]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T13:11:04.055 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1711026662,"num_in_osds":8,"osd_in_since":1711026644,"num_remapped_pgs":1} 2024-03-21T13:11:04.056 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T13:11:04.295 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: Detected new or changed devices on smithi152 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi152 to 4272M 2024-03-21T13:11:05.518 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.518 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:05.518 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:05.518 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.518 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T13:11:05.518 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[30095]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: Detected new or changed devices on smithi152 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi152 to 4272M 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.519 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T13:11:05.520 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:05 smithi089 ceph-mon[39209]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:05.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: Detected new or changed devices on smithi152 2024-03-21T13:11:05.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi152 to 4272M 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T13:11:05.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:05 smithi152 ceph-mon[34051]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 -- 172.21.15.89:0/3900308377 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1021e0 msgr2=0x7f8d3c102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3900308377 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1021e0 0x7f8d3c102660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d300098f0 tx=0x7f8d30009c00 comp rx=0 tx=0).stop 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 -- 172.21.15.89:0/3900308377 shutdown_connections 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3900308377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 0x7f8d3c10efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3900308377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d3c1081c0 0x7f8d3c1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3900308377 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1021e0 0x7f8d3c102660 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 -- 172.21.15.89:0/3900308377 >> 172.21.15.89:0/3900308377 conn(0x7f8d3c0792c0 msgr2=0x7f8d3c0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 -- 172.21.15.89:0/3900308377 shutdown_connections 2024-03-21T13:11:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 -- 172.21.15.89:0/3900308377 wait complete. 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 Processor -- start 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.880+0000 7f8d409a6700 1 -- start start 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d3c1021e0 0x7f8d3c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 0x7f8d3c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1081c0 0x7f8d3c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c1118f0 con 0x7f8d3c102ba0 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c111770 con 0x7f8d3c1021e0 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c111a70 con 0x7f8d3c1081c0 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 0x7f8d3c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 0x7f8d3c198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42330/0 (socket says 172.21.15.89:42330) 2024-03-21T13:11:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 -- 172.21.15.89:0/3319055418 learned_addr learned my addr 172.21.15.89:0/3319055418 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d3ad9d700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1081c0 0x7f8d3c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d3a59c700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d3c1021e0 0x7f8d3c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 -- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1081c0 msgr2=0x7f8d3c19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1081c0 0x7f8d3c19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 -- 172.21.15.89:0/3319055418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d3c1021e0 msgr2=0x7f8d3c197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d3c1021e0 0x7f8d3c197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 -- 172.21.15.89:0/3319055418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c19ce40 con 0x7f8d3c102ba0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d39d9b700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 0x7f8d3c198370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000fa70 tx=0x7f8d3000faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d2b7fe700 1 -- 172.21.15.89:0/3319055418 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30027070 con 0x7f8d3c102ba0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d2b7fe700 1 -- 172.21.15.89:0/3319055418 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8d3000bb60 con 0x7f8d3c102ba0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c19d0d0 con 0x7f8d3c102ba0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d2b7fe700 1 -- 172.21.15.89:0/3319055418 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30004930 con 0x7f8d3c102ba0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.881+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c1a49b0 con 0x7f8d3c102ba0 2024-03-21T13:11:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.882+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c068e50 con 0x7f8d3c102ba0 2024-03-21T13:11:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.883+0000 7f8d2b7fe700 1 -- 172.21.15.89:0/3319055418 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7f8d3002a070 con 0x7f8d3c102ba0 2024-03-21T13:11:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.883+0000 7f8d2b7fe700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8d2406e250 0x7f8d24070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.883+0000 7f8d2b7fe700 1 -- 172.21.15.89:0/3319055418 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f8d3008e9f0 con 0x7f8d3c102ba0 2024-03-21T13:11:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.883+0000 7f8d3a59c700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8d2406e250 0x7f8d24070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.884+0000 7f8d3a59c700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8d2406e250 0x7f8d24070710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c005950 tx=0x7f8d2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:05.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:05.888+0000 7f8d2b7fe700 1 -- 172.21.15.89:0/3319055418 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d30013030 con 0x7f8d3c102ba0 2024-03-21T13:11:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.085+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d3c1a4b80 con 0x7f8d3c102ba0 2024-03-21T13:11:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.086+0000 7f8d2b7fe700 1 -- 172.21.15.89:0/3319055418 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13551 (secure 0 0 0) 0x7f8d3005c510 con 0x7f8d3c102ba0 2024-03-21T13:11:06.087 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:11:06.087 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":59,"fsid":"9c355834-e783-11ee-95cd-87774f69a715","created":"2024-03-21T13:05:40.974140+0000","modified":"2024-03-21T13:11:05.383632+0000","last_up_change":"2024-03-21T13:11:02.379434+0000","last_in_change":"2024-03-21T13:10:44.727962+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T13:09:01.242029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4258f99e-44cf-4cd0-a093-00e1192a8ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6803","nonce":1699462714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6805","nonce":1699462714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6809","nonce":1699462714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6807","nonce":1699462714}]},"public_addr":"172.21.15.89:6803/1699462714","cluster_addr":"172.21.15.89:6805/1699462714","heartbeat_back_addr":"172.21.15.89:6809/1699462714","heartbeat_front_addr":"172.21.15.89:6807/1699462714","state":["exists","up"]},{"osd":1,"uuid":"debf8a73-c369-49ed-8886-57f2ca61e6d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6811","nonce":3822115704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6813","nonce":3822115704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6817","nonce":3822115704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6815","nonce":3822115704}]},"public_addr":"172.21.15.89:6811/3822115704","cluster_addr":"172.21.15.89:6813/3822115704","heartbeat_back_addr":"172.21.15.89:6817/3822115704","heartbeat_front_addr":"172.21.15.89:6815/3822115704","state":["exists","up"]},{"osd":2,"uuid":"2686770f-bbbc-4082-bb00-9f6f10f8783b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6819","nonce":2934798428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6821","nonce":2934798428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6825","nonce":2934798428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6823","nonce":2934798428}]},"public_addr":"172.21.15.89:6819/2934798428","cluster_addr":"172.21.15.89:6821/2934798428","heartbeat_back_addr":"172.21.15.89:6825/2934798428","heartbeat_front_addr":"172.21.15.89:6823/2934798428","state":["exists","up"]},{"osd":3,"uuid":"4c996867-ce38-4ae7-b934-5faf2ce2fc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6827","nonce":193784582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6829","nonce":193784582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6833","nonce":193784582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6831","nonce":193784582}]},"public_addr":"172.21.15.89:6827/193784582","cluster_addr":"172.21.15.89:6829/193784582","heartbeat_back_addr":"172.21.15.89:6833/193784582","heartbeat_front_addr":"172.21.15.89:6831/193784582","state":["exists","up"]},{"osd":4,"uuid":"3d743da7-8b8c-40f8-a591-e791ed280233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6801","nonce":2252846533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6803","nonce":2252846533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6807","nonce":2252846533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6805","nonce":2252846533}]},"public_addr":"172.21.15.152:6801/2252846533","cluster_addr":"172.21.15.152:6803/2252846533","heartbeat_back_addr":"172.21.15.152:6807/2252846533","heartbeat_front_addr":"172.21.15.152:6805/2252846533","state":["exists","up"]},{"osd":5,"uuid":"e2ece7d8-4a4a-4247-b3af-46c854bcc1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6809","nonce":3145783803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6811","nonce":3145783803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6815","nonce":3145783803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6813","nonce":3145783803}]},"public_addr":"172.21.15.152:6809/3145783803","cluster_addr":"172.21.15.152:6811/3145783803","heartbeat_back_addr":"172.21.15.152:6815/3145783803","heartbeat_front_addr":"172.21.15.152:6813/3145783803","state":["exists","up"]},{"osd":6,"uuid":"877ec6ea-e165-4d53-aa88-87004c5b26c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6817","nonce":3295577655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6819","nonce":3295577655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6823","nonce":3295577655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6821","nonce":3295577655}]},"public_addr":"172.21.15.152:6817/3295577655","cluster_addr":"172.21.15.152:6819/3295577655","heartbeat_back_addr":"172.21.15.152:6823/3295577655","heartbeat_front_addr":"172.21.15.152:6821/3295577655","state":["exists","up"]},{"osd":7,"uuid":"787eb11f-c37b-4862-b1e7-485a9da2851e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6825","nonce":440724144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6827","nonce":440724144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6831","nonce":440724144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6829","nonce":440724144}]},"public_addr":"172.21.15.152:6825/440724144","cluster_addr":"172.21.15.152:6827/440724144","heartbeat_back_addr":"172.21.15.152:6831/440724144","heartbeat_front_addr":"172.21.15.152:6829/440724144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:07.805982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:32.484356+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:57.086765+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:09:21.879472+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:09:44.830909+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:10:10.422590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:10:35.568456+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:11:00.893882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/745558169":"2024-03-22T13:06:42.905029+0000","172.21.15.89:6801/1022748022":"2024-03-22T13:06:42.905029+0000","172.21.15.89:6800/1022748022":"2024-03-22T13:06:42.905029+0000","172.21.15.89:0/3750854419":"2024-03-22T13:06:16.698081+0000","172.21.15.89:6800/3099145381":"2024-03-22T13:06:16.698081+0000","172.21.15.89:6801/3099145381":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/1302626015":"2024-03-22T13:06:42.905029+0000","172.21.15.89:0/346547163":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/249140509":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/586007047":"2024-03-22T13:06:42.905029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8d2406e250 msgr2=0x7f8d24070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8d2406e250 0x7f8d24070710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c005950 tx=0x7f8d2c00b410 comp rx=0 tx=0).stop 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 msgr2=0x7f8d3c198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 0x7f8d3c198370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000fa70 tx=0x7f8d3000faa0 comp rx=0 tx=0).stop 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 shutdown_connections 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8d3c1081c0 0x7f8d3c19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f8d2406e250 0x7f8d24070710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d3c1021e0 0x7f8d3c197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 --2- 172.21.15.89:0/3319055418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d3c102ba0 0x7f8d3c198370 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 >> 172.21.15.89:0/3319055418 conn(0x7f8d3c0792c0 msgr2=0x7f8d3c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 shutdown_connections 2024-03-21T13:11:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:06.088+0000 7f8d409a6700 1 -- 172.21.15.89:0/3319055418 wait complete. 2024-03-21T13:11:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:06 smithi152 ceph-mon[34051]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T13:11:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3319055418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:06.631 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:06 smithi089 ceph-mon[30095]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T13:11:06.631 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3319055418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:06.632 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:06 smithi089 ceph-mon[39209]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T13:11:06.632 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3319055418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:06.660 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T13:09:01.242029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T13:11:06.660 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-21T13:11:06.897 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:07.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:07 smithi152 ceph-mon[34051]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T13:11:07.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:07 smithi152 ceph-mon[34051]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:07 smithi089 ceph-mon[30095]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T13:11:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:07 smithi089 ceph-mon[30095]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:07 smithi089 ceph-mon[39209]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T13:11:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:07 smithi089 ceph-mon[39209]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:08.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/2558469707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf8102ba0 msgr2=0x7f2cf810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/2558469707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf8102ba0 0x7f2cf810efe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000b0d0 tx=0x7f2cf000b3e0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/2558469707 shutdown_connections 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/2558469707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf8102ba0 0x7f2cf810efe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/2558469707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cf81081c0 0x7f2cf81085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/2558469707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 0x7f2cf8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/2558469707 >> 172.21.15.89:0/2558469707 conn(0x7f2cf80792c0 msgr2=0x7f2cf80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/2558469707 shutdown_connections 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.591+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/2558469707 wait complete. 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 Processor -- start 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 -- start start 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 0x7f2cf819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cf8102ba0 0x7f2cf819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81081c0 0x7f2cf8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf81118f0 con 0x7f2cf81081c0 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf8111770 con 0x7f2cf8102ba0 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfe3d4700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf8111a70 con 0x7f2cf81021e0 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 0x7f2cf819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 0x7f2cf819da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:57696/0 (socket says 172.21.15.89:57696) 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 -- 172.21.15.89:0/1563183490 learned_addr learned my addr 172.21.15.89:0/1563183490 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfc971700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81081c0 0x7f2cf8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf77fe700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cf8102ba0 0x7f2cf819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 -- 172.21.15.89:0/1563183490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cf8102ba0 msgr2=0x7f2cf819df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cf8102ba0 0x7f2cf819df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 -- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81081c0 msgr2=0x7f2cf8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81081c0 0x7f2cf8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cf7fff700 1 -- 172.21.15.89:0/1563183490 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8198440 con 0x7f2cf81021e0 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.592+0000 7f2cfc971700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81081c0 0x7f2cf8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.593+0000 7f2cf7fff700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 0x7f2cf819da40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00d950 tx=0x7f2cec00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.593+0000 7f2cf57fa700 1 -- 172.21.15.89:0/1563183490 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec016070 con 0x7f2cf81021e0 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.593+0000 7f2cf57fa700 1 -- 172.21.15.89:0/1563183490 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2cec00fd10 con 0x7f2cf81021e0 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.593+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf8198730 con 0x7f2cf81021e0 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.593+0000 7f2cf57fa700 1 -- 172.21.15.89:0/1563183490 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00e3b0 con 0x7f2cf81021e0 2024-03-21T13:11:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.593+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf81a49c0 con 0x7f2cf81021e0 2024-03-21T13:11:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.594+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf8103870 con 0x7f2cf81021e0 2024-03-21T13:11:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.594+0000 7f2cf57fa700 1 -- 172.21.15.89:0/1563183490 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7f2cec019070 con 0x7f2cf81021e0 2024-03-21T13:11:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.595+0000 7f2cf57fa700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f2ce006e250 0x7f2ce0070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.595+0000 7f2cf57fa700 1 -- 172.21.15.89:0/1563183490 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f2cec08d9e0 con 0x7f2cf81021e0 2024-03-21T13:11:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.595+0000 7f2cf77fe700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f2ce006e250 0x7f2ce0070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.595+0000 7f2cf77fe700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f2ce006e250 0x7f2ce0070710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009680 tx=0x7f2ce8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.600+0000 7f2cf57fa700 1 -- 172.21.15.89:0/1563183490 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2cec05ef50 con 0x7f2cf81021e0 2024-03-21T13:11:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.802+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2cf8198960 con 0x7f2cf81021e0 2024-03-21T13:11:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.803+0000 7f2cf57fa700 1 -- 172.21.15.89:0/1563183490 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2cec063780 con 0x7f2cf81021e0 2024-03-21T13:11:08.804 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-03-21T13:11:08.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f2ce006e250 msgr2=0x7f2ce0070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:08.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f2ce006e250 0x7f2ce0070710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009680 tx=0x7f2ce8009260 comp rx=0 tx=0).stop 2024-03-21T13:11:08.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 msgr2=0x7f2cf819da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:08.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 0x7f2cf819da40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00d950 tx=0x7f2cec00dc60 comp rx=0 tx=0).stop 2024-03-21T13:11:08.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 shutdown_connections 2024-03-21T13:11:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81081c0 0x7f2cf8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2cf81021e0 0x7f2cf819da40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f2ce006e250 0x7f2ce0070710 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 --2- 172.21.15.89:0/1563183490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cf8102ba0 0x7f2cf819df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.805+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 >> 172.21.15.89:0/1563183490 conn(0x7f2cf80792c0 msgr2=0x7f2cf8105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.806+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 shutdown_connections 2024-03-21T13:11:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:08.806+0000 7f2cfe3d4700 1 -- 172.21.15.89:0/1563183490 wait complete. 2024-03-21T13:11:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1563183490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T13:11:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1563183490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T13:11:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1563183490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T13:11:09.491 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi089 2024-03-21T13:11:09.492 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi089=foo.a' 2024-03-21T13:11:09.746 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:10.126 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:09 smithi152 ceph-mon[34051]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T13:11:10.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:09 smithi089 ceph-mon[39209]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T13:11:10.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:09 smithi089 ceph-mon[30095]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T13:11:11.400 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 -- 172.21.15.152:0/4135113434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640100ae0 msgr2=0x7fc640100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:11.400 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 --2- 172.21.15.152:0/4135113434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640100ae0 0x7fc640100f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc630009990 tx=0x7fc630009ca0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.400 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 -- 172.21.15.152:0/4135113434 shutdown_connections 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 --2- 172.21.15.152:0/4135113434 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc640101e00 0x7fc640102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 --2- 172.21.15.152:0/4135113434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640100ae0 0x7fc640100f00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 --2- 172.21.15.152:0/4135113434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc640101440 0x7fc6401018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 -- 172.21.15.152:0/4135113434 >> 172.21.15.152:0/4135113434 conn(0x7fc6400fc720 msgr2=0x7fc6400feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 -- 172.21.15.152:0/4135113434 shutdown_connections 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.397+0000 7fc64660c700 1 -- 172.21.15.152:0/4135113434 wait complete. 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 Processor -- start 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 -- start start 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc640100ae0 0x7fc640199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640101440 0x7fc64019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc640101e00 0x7fc6401a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64010f480 con 0x7fc640100ae0 2024-03-21T13:11:11.401 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64010f300 con 0x7fc640101440 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc64660c700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64010f600 con 0x7fc640101e00 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc63f7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640101440 0x7fc64019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc63f7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640101440 0x7fc64019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:52172/0 (socket says 172.21.15.152:52172) 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc63f7fe700 1 -- 172.21.15.152:0/3865957117 learned_addr learned my addr 172.21.15.152:0/3865957117 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.398+0000 7fc63ffff700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc640100ae0 0x7fc640199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc644ba9700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc640101e00 0x7fc6401a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63f7fe700 1 -- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc640101e00 msgr2=0x7fc6401a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63f7fe700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc640101e00 0x7fc6401a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63f7fe700 1 -- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc640100ae0 msgr2=0x7fc640199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63f7fe700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc640100ae0 0x7fc640199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63f7fe700 1 -- 172.21.15.152:0/3865957117 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6401a1720 con 0x7fc640101440 2024-03-21T13:11:11.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63f7fe700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640101440 0x7fc64019a000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc63400c9b0 tx=0x7fc63400ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:11.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63d7fa700 1 -- 172.21.15.152:0/3865957117 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634007b00 con 0x7fc640101440 2024-03-21T13:11:11.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6401a1a10 con 0x7fc640101440 2024-03-21T13:11:11.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63d7fa700 1 -- 172.21.15.152:0/3865957117 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc634007ca0 con 0x7fc640101440 2024-03-21T13:11:11.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc63d7fa700 1 -- 172.21.15.152:0/3865957117 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63400fe60 con 0x7fc640101440 2024-03-21T13:11:11.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.399+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6401a1fa0 con 0x7fc640101440 2024-03-21T13:11:11.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.400+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc640068e50 con 0x7fc640101440 2024-03-21T13:11:11.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.400+0000 7fc63d7fa700 1 -- 172.21.15.152:0/3865957117 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7fc63400f450 con 0x7fc640101440 2024-03-21T13:11:11.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.401+0000 7fc63d7fa700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc62806e250 0x7fc628070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:11.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.401+0000 7fc63d7fa700 1 -- 172.21.15.152:0/3865957117 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fc63408dee0 con 0x7fc640101440 2024-03-21T13:11:11.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.401+0000 7fc63ffff700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc62806e250 0x7fc628070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:11.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.402+0000 7fc63ffff700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc62806e250 0x7fc628070710 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc63000b5c0 tx=0x7fc6300036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:11.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.406+0000 7fc63d7fa700 1 -- 172.21.15.152:0/3865957117 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc634014030 con 0x7fc640101440 2024-03-21T13:11:11.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.615+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi089=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fc64002dcf0 con 0x7fc62806e250 2024-03-21T13:11:11.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.622+0000 7fc63d7fa700 1 -- 172.21.15.152:0/3865957117 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7fc64002dcf0 con 0x7fc62806e250 2024-03-21T13:11:11.622 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled rgw.foo.a update... 2024-03-21T13:11:11.624 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc62806e250 msgr2=0x7fc628070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc62806e250 0x7fc628070710 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc63000b5c0 tx=0x7fc6300036f0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640101440 msgr2=0x7fc64019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640101440 0x7fc64019a000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc63400c9b0 tx=0x7fc63400ccc0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 shutdown_connections 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc640101e00 0x7fc6401a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc640100ae0 0x7fc640199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fc62806e250 0x7fc628070710 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 --2- 172.21.15.152:0/3865957117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc640101440 0x7fc64019a000 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 >> 172.21.15.152:0/3865957117 conn(0x7fc6400fc720 msgr2=0x7fc6400fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 shutdown_connections 2024-03-21T13:11:11.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:11.624+0000 7fc64660c700 1 -- 172.21.15.152:0/3865957117 wait complete. 2024-03-21T13:11:12.179 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:12 smithi152 ceph-mon[34051]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-21T13:11:12.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:12.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:12 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:12.208 DEBUG:teuthology.orchestra.run.smithi089:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-21T13:11:12.211 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi152 2024-03-21T13:11:12.211 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-21T13:11:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:12 smithi089 ceph-mon[30095]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-21T13:11:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:12 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:12 smithi089 ceph-mon[39209]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-21T13:11:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:12 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:12.445 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi089=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: Saving service rgw.foo.a spec with placement smithi089=foo.a;count:1 2024-03-21T13:11:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:11:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T13:11:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:13 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi089=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: Saving service rgw.foo.a spec with placement smithi089=foo.a;count:1 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:13.194 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi089=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: Saving service rgw.foo.a spec with placement smithi089=foo.a;count:1 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:13.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:13 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:14.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 -- 172.21.15.152:0/3248841978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103400 msgr2=0x7f68cc103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:14.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/3248841978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103400 0x7f68cc103860 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68c00099c0 tx=0x7f68c0009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 -- 172.21.15.152:0/3248841978 shutdown_connections 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/3248841978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68cc103da0 0x7f68cc10a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/3248841978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68cc102200 0x7f68cc102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/3248841978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103400 0x7f68cc103860 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 -- 172.21.15.152:0/3248841978 >> 172.21.15.152:0/3248841978 conn(0x7f68cc0fd7e0 msgr2=0x7f68cc0ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 -- 172.21.15.152:0/3248841978 shutdown_connections 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 -- 172.21.15.152:0/3248841978 wait complete. 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.015+0000 7f68d3c5f700 1 Processor -- start 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 -- start start 2024-03-21T13:11:14.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68cc102200 0x7f68cc197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68cc103400 0x7f68cc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103da0 0x7f68cc19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc10d320 con 0x7f68cc103400 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc10d1a0 con 0x7f68cc103da0 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc10d4a0 con 0x7f68cc102200 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103da0 0x7f68cc19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103da0 0x7f68cc19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:52178/0 (socket says 172.21.15.152:52178) 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 -- 172.21.15.152:0/1456803074 learned_addr learned my addr 172.21.15.152:0/1456803074 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d19fb700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68cc102200 0x7f68cc197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d11fa700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68cc103400 0x7f68cc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:14.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 -- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68cc102200 msgr2=0x7f68cc197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68cc102200 0x7f68cc197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 -- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68cc103400 msgr2=0x7f68cc197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68cc103400 0x7f68cc197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 -- 172.21.15.152:0/1456803074 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc19f5d0 con 0x7f68cc103da0 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d21fc700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103da0 0x7f68cc19ef30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00d980 tx=0x7f68bc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68caffd700 1 -- 172.21.15.152:0/1456803074 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc016070 con 0x7f68cc103da0 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68caffd700 1 -- 172.21.15.152:0/1456803074 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f68bc004830 con 0x7f68cc103da0 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.016+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc19f8c0 con 0x7f68cc103da0 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.017+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc19fe50 con 0x7f68cc103da0 2024-03-21T13:11:14.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.017+0000 7f68caffd700 1 -- 172.21.15.152:0/1456803074 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc005400 con 0x7f68cc103da0 2024-03-21T13:11:14.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.018+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc068e50 con 0x7f68cc103da0 2024-03-21T13:11:14.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.018+0000 7f68caffd700 1 -- 172.21.15.152:0/1456803074 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7f68bc0055a0 con 0x7f68cc103da0 2024-03-21T13:11:14.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.018+0000 7f68caffd700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f68b406e250 0x7f68b4070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:14.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.018+0000 7f68caffd700 1 -- 172.21.15.152:0/1456803074 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f68bc08db40 con 0x7f68cc103da0 2024-03-21T13:11:14.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.019+0000 7f68d19fb700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f68b406e250 0x7f68b4070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:14.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.019+0000 7f68d19fb700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f68b406e250 0x7f68b4070710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009550 tx=0x7f68c40091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:14.023 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.023+0000 7f68caffd700 1 -- 172.21.15.152:0/1456803074 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f68bc01f020 con 0x7f68cc103da0 2024-03-21T13:11:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:14 smithi152 ceph-mon[34051]: Deploying daemon rgw.foo.a on smithi089 2024-03-21T13:11:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:14 smithi152 ceph-mon[34051]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T13:11:14.196 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:14 smithi089 ceph-mon[39209]: Deploying daemon rgw.foo.a on smithi089 2024-03-21T13:11:14.196 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:14 smithi089 ceph-mon[39209]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T13:11:14.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:14 smithi089 ceph-mon[30095]: Deploying daemon rgw.foo.a on smithi089 2024-03-21T13:11:14.197 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:14 smithi089 ceph-mon[30095]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T13:11:14.219 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:14.218+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f68cc02da30 con 0x7f68cc103da0 2024-03-21T13:11:14.575 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:11:14 smithi089 systemd[1]: Starting Ceph rgw.foo.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:11:14.843 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:11:14 smithi089 podman[67497]: 2024-03-21 13:11:14.493893066 +0000 UTC m=+0.017777002 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:11:14.844 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:11:14 smithi089 podman[67497]: 2024-03-21 13:11:14.609541228 +0000 UTC m=+0.133425158 container create d1ba0a72303fe78dfeb01dfa4c72af62c6f56e1f6547e7a9b5176a90db6e9e68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-21T13:11:14.844 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:11:14 smithi089 podman[67497]: 2024-03-21 13:11:14.837938005 +0000 UTC m=+0.361821940 container init d1ba0a72303fe78dfeb01dfa4c72af62c6f56e1f6547e7a9b5176a90db6e9e68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T13:11:14.844 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:11:14 smithi089 podman[67497]: 2024-03-21 13:11:14.84353821 +0000 UTC m=+0.367422141 container start d1ba0a72303fe78dfeb01dfa4c72af62c6f56e1f6547e7a9b5176a90db6e9e68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T13:11:15.120 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:11:14 smithi089 bash[67497]: d1ba0a72303fe78dfeb01dfa4c72af62c6f56e1f6547e7a9b5176a90db6e9e68 2024-03-21T13:11:15.120 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:11:14 smithi089 systemd[1]: Started Ceph rgw.foo.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:11:15.135 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.134+0000 7f68caffd700 1 -- 172.21.15.152:0/1456803074 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v61) v1 ==== 160+0+0 (secure 0 0 0) 0x7f68bc022350 con 0x7f68cc103da0 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f68b406e250 msgr2=0x7f68b4070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f68b406e250 0x7f68b4070710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009550 tx=0x7f68c40091b0 comp rx=0 tx=0).stop 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103da0 msgr2=0x7f68cc19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103da0 0x7f68cc19ef30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00d980 tx=0x7f68bc00dc90 comp rx=0 tx=0).stop 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 shutdown_connections 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f68b406e250 0x7f68b4070710 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68cc103da0 0x7f68cc19ef30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68cc102200 0x7f68cc197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 --2- 172.21.15.152:0/1456803074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68cc103400 0x7f68cc197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 >> 172.21.15.152:0/1456803074 conn(0x7f68cc0fd7e0 msgr2=0x7f68cc106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:15.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 shutdown_connections 2024-03-21T13:11:15.141 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:15.139+0000 7f68d3c5f700 1 -- 172.21.15.152:0/1456803074 wait complete. 2024-03-21T13:11:15.141 INFO:teuthology.orchestra.run.smithi152.stderr:pool 'datapool' created 2024-03-21T13:11:15.470 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/1456803074' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T13:11:15.470 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T13:11:15.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/1456803074' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T13:11:15.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T13:11:15.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/1456803074' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T13:11:15.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T13:11:15.784 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- rbd pool init datapool 2024-03-21T13:11:16.022 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:16.449 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-21T13:11:16.449 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T13:11:16.449 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.449 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T13:11:16.449 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3249261794' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T13:11:16.449 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T13:11:16.449 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.450 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.450 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: Saving service rgw.foo.a spec with placement smithi089=foo.a;count:1 2024-03-21T13:11:16.450 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.450 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.450 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:16 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-21T13:11:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T13:11:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T13:11:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3249261794' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T13:11:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T13:11:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: Saving service rgw.foo.a spec with placement smithi089=foo.a;count:1 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3249261794' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T13:11:16.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: Saving service rgw.foo.a spec with placement smithi089=foo.a;count:1 2024-03-21T13:11:16.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:16.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:16 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:17.390 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T13:11:17.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:17 smithi152 ceph-mon[34051]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T13:11:17.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:17.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:17.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:17.391 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:17 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:17.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T13:11:17.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[30095]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T13:11:17.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:17.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:17.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:17.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:17.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T13:11:17.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[39209]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T13:11:17.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:17.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:17.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:17.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:17 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: Checking dashboard <-> RGW credentials 2024-03-21T13:11:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: pgmap v189: 36 pgs: 6 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T13:11:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T13:11:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T13:11:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/3292946823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T13:11:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T13:11:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:11:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:18 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: Checking dashboard <-> RGW credentials 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: pgmap v189: 36 pgs: 6 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/3292946823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: Checking dashboard <-> RGW credentials 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: pgmap v189: 36 pgs: 6 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/3292946823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:11:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:18 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T13:11:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T13:11:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T13:11:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[30095]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T13:11:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[30095]: pgmap v192: 68 pgs: 15 creating+peering, 20 active+clean, 33 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:11:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T13:11:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T13:11:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T13:11:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[39209]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T13:11:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:19 smithi089 ceph-mon[39209]: pgmap v192: 68 pgs: 15 creating+peering, 20 active+clean, 33 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:19 smithi152 ceph-mon[34051]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T13:11:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:19 smithi152 ceph-mon[34051]: pgmap v192: 68 pgs: 15 creating+peering, 20 active+clean, 33 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:11:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:20 smithi152 ceph-mon[34051]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:11:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:20 smithi152 ceph-mon[34051]: Cluster is now healthy 2024-03-21T13:11:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:20 smithi152 ceph-mon[34051]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T13:11:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T13:11:20.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T13:11:20.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[30095]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[30095]: Cluster is now healthy 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[30095]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[39209]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[39209]: Cluster is now healthy 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[39209]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T13:11:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T13:11:20.958 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.152 --placement '1;smithi152=iscsi.a' 2024-03-21T13:11:21.199 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:21.556 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T13:11:21.556 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T13:11:21.556 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:21 smithi152 ceph-mon[34051]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T13:11:21.556 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:21 smithi152 ceph-mon[34051]: pgmap v195: 100 pgs: 18 creating+peering, 57 active+clean, 25 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T13:11:21.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T13:11:21.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T13:11:21.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[30095]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T13:11:21.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[30095]: pgmap v195: 100 pgs: 18 creating+peering, 57 active+clean, 25 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T13:11:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T13:11:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T13:11:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[39209]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T13:11:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:21 smithi089 ceph-mon[39209]: pgmap v195: 100 pgs: 18 creating+peering, 57 active+clean, 25 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T13:11:22.560 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:22 smithi152 ceph-mon[34051]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T13:11:22.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T13:11:22.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T13:11:22.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:22 smithi089 ceph-mon[30095]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T13:11:22.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T13:11:22.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T13:11:22.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:22 smithi089 ceph-mon[39209]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T13:11:22.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T13:11:22.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T13:11:22.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.781+0000 7fac07492700 1 -- 172.21.15.152:0/3560850506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00102d20 msgr2=0x7fac00105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.781+0000 7fac07492700 1 --2- 172.21.15.152:0/3560850506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00102d20 0x7fac00105110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009990 tx=0x7fabfc009ca0 comp rx=0 tx=0).stop 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- 172.21.15.152:0/3560850506 shutdown_connections 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 --2- 172.21.15.152:0/3560850506 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac00107f80 0x7fac0010a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 --2- 172.21.15.152:0/3560850506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00102d20 0x7fac00105110 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 --2- 172.21.15.152:0/3560850506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac00105650 0x7fac00107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- 172.21.15.152:0/3560850506 >> 172.21.15.152:0/3560850506 conn(0x7fac000fc6b0 msgr2=0x7fac000fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- 172.21.15.152:0/3560850506 shutdown_connections 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- 172.21.15.152:0/3560850506 wait complete. 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 Processor -- start 2024-03-21T13:11:22.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- start start 2024-03-21T13:11:22.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac00102d20 0x7fac00102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:22.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac00105650 0x7fac00100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:22.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00107f80 0x7fac00101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0010d270 con 0x7fac00105650 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0010d0f0 con 0x7fac00107f80 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac07492700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0010d3f0 con 0x7fac00102d20 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.782+0000 7fac05a2f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00107f80 0x7fac00101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac0522e700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac00102d20 0x7fac00102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00107f80 0x7fac00101210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:36786/0 (socket says 172.21.15.152:36786) 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac0522e700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac00102d20 0x7fac00102680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.152:45646/0 (socket says 172.21.15.152:45646) 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 -- 172.21.15.152:0/1304942344 learned_addr learned my addr 172.21.15.152:0/1304942344 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac04a2d700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac00105650 0x7fac00100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 -- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac00102d20 msgr2=0x7fac00102680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:22.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac00102d20 0x7fac00102680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 -- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac00105650 msgr2=0x7fac00100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac00105650 0x7fac00100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 -- 172.21.15.152:0/1304942344 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac00101ac0 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac05a2f700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00107f80 0x7fac00101210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fabf8007d60 tx=0x7fabf800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fabf67fc700 1 -- 172.21.15.152:0/1304942344 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf800ac90 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac001a19a0 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fabf67fc700 1 -- 172.21.15.152:0/1304942344 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fabf800ae30 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fabf67fc700 1 -- 172.21.15.152:0/1304942344 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8004790 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.783+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac001a1e60 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.784+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac000506a0 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.785+0000 7fabf67fc700 1 -- 172.21.15.152:0/1304942344 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7fabf8003a00 con 0x7fac00107f80 2024-03-21T13:11:22.787 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.785+0000 7fabf67fc700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fabec06e2f0 0x7fabec0707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:22.788 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.785+0000 7fabf67fc700 1 -- 172.21.15.152:0/1304942344 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7132+0+0 (secure 0 0 0) 0x7fabf808e790 con 0x7fac00107f80 2024-03-21T13:11:22.788 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.785+0000 7fac0522e700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fabec06e2f0 0x7fabec0707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:22.788 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.786+0000 7fac0522e700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fabec06e2f0 0x7fabec0707b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00b5c0 tx=0x7fabfc0036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:22.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.790+0000 7fabf67fc700 1 -- 172.21.15.152:0/1304942344 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fabf805bb80 con 0x7fac00107f80 2024-03-21T13:11:22.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:22.997+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.152", "placement": "1;smithi152=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fac000625d0 con 0x7fabec06e2f0 2024-03-21T13:11:23.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.006+0000 7fabf67fc700 1 -- 172.21.15.152:0/1304942344 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7fac000625d0 con 0x7fabec06e2f0 2024-03-21T13:11:23.006 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled iscsi.datapool update... 2024-03-21T13:11:23.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fabec06e2f0 msgr2=0x7fabec0707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fabec06e2f0 0x7fabec0707b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00b5c0 tx=0x7fabfc0036f0 comp rx=0 tx=0).stop 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00107f80 msgr2=0x7fac00101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00107f80 0x7fac00101210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fabf8007d60 tx=0x7fabf800a340 comp rx=0 tx=0).stop 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 shutdown_connections 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fabec06e2f0 0x7fabec0707b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac00107f80 0x7fac00101210 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac00102d20 0x7fac00102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 --2- 172.21.15.152:0/1304942344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac00105650 0x7fac00100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 >> 172.21.15.152:0/1304942344 conn(0x7fac000fc6b0 msgr2=0x7fac00103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 shutdown_connections 2024-03-21T13:11:23.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:23.008+0000 7fac07492700 1 -- 172.21.15.152:0/1304942344 wait complete. 2024-03-21T13:11:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T13:11:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T13:11:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T13:11:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T13:11:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T13:11:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: pgmap v198: 132 pgs: 3 creating+activating, 13 creating+peering, 95 active+clean, 21 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:11:23.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.152", "placement": "1;smithi152=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:23.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: Saving service iscsi.datapool spec with placement smithi152=iscsi.a;count:1 2024-03-21T13:11:23.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:23.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T13:11:23.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T13:11:23.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:23 smithi152 ceph-mon[34051]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: pgmap v198: 132 pgs: 3 creating+activating, 13 creating+peering, 95 active+clean, 21 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.152", "placement": "1;smithi152=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: Saving service iscsi.datapool spec with placement smithi152=iscsi.a;count:1 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T13:11:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[30095]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: pgmap v198: 132 pgs: 3 creating+activating, 13 creating+peering, 95 active+clean, 21 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.152", "placement": "1;smithi152=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: Saving service iscsi.datapool spec with placement smithi152=iscsi.a;count:1 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:23.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/533721271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T13:11:23.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2223002524' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T13:11:23.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:23 smithi089 ceph-mon[39209]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T13:11:23.659 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-21T13:11:23.660 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:11:23.660 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-21T13:11:23.692 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:11:23.692 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-21T13:11:23.721 DEBUG:teuthology.orchestra.run.smithi152:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@iscsi.iscsi.a.service 2024-03-21T13:11:23.764 INFO:tasks.cephadm:Adding prometheus.a on smithi152 2024-03-21T13:11:23.764 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi152=a' 2024-03-21T13:11:24.053 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:24.586 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.587 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.587 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.587 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:11:24.587 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T13:11:24.587 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:24 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:24.587 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:24 smithi152 ceph-mon[34051]: Deploying daemon iscsi.iscsi.a on smithi152 2024-03-21T13:11:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:11:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T13:11:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[30095]: Deploying daemon iscsi.iscsi.a on smithi152 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:24.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:24 smithi089 ceph-mon[39209]: Deploying daemon iscsi.iscsi.a on smithi152 2024-03-21T13:11:25.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:25 smithi152 ceph-mon[34051]: pgmap v200: 132 pgs: 3 creating+activating, 13 creating+peering, 106 active+clean, 10 unknown; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.4 KiB/s wr, 50 op/s 2024-03-21T13:11:25.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:25 smithi089 ceph-mon[39209]: pgmap v200: 132 pgs: 3 creating+activating, 13 creating+peering, 106 active+clean, 10 unknown; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.4 KiB/s wr, 50 op/s 2024-03-21T13:11:25.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:25 smithi089 ceph-mon[30095]: pgmap v200: 132 pgs: 3 creating+activating, 13 creating+peering, 106 active+clean, 10 unknown; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.4 KiB/s wr, 50 op/s 2024-03-21T13:11:26.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:26 smithi152 systemd[1]: Starting Ceph iscsi.iscsi.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:11:27.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/3975446042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc102c90 msgr2=0x7f0abc109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:27.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/3975446042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc102c90 0x7f0abc109540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400b0d0 tx=0x7f0ab400b3e0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/3975446042 shutdown_connections 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/3975446042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc102c90 0x7f0abc109540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/3975446042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0abc1010f0 0x7f0abc101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/3975446042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0abc1022f0 0x7f0abc102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/3975446042 >> 172.21.15.152:0/3975446042 conn(0x7f0abc0fc690 msgr2=0x7f0abc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/3975446042 shutdown_connections 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:26.999+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/3975446042 wait complete. 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.000+0000 7f0ac1ca6700 1 Processor -- start 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.000+0000 7f0ac1ca6700 1 -- start start 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0ac1ca6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc1010f0 0x7f0abc1967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:27.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0ac1ca6700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0abc1022f0 0x7f0abc196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0ac1ca6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0abc102c90 0x7f0abc19dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0ac1ca6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc10c1e0 con 0x7f0abc102c90 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0ac1ca6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc10c060 con 0x7f0abc1010f0 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0ac1ca6700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc10c360 con 0x7f0abc1022f0 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0abc102c90 0x7f0abc19dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc1010f0 0x7f0abc1967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0abc1022f0 0x7f0abc196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc1010f0 0x7f0abc1967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:43790/0 (socket says 172.21.15.152:43790) 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abb7fe700 1 -- 172.21.15.152:0/359840158 learned_addr learned my addr 172.21.15.152:0/359840158 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abaffd700 1 -- 172.21.15.152:0/359840158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc1010f0 msgr2=0x7f0abc1967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abaffd700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc1010f0 0x7f0abc1967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abaffd700 1 -- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0abc102c90 msgr2=0x7f0abc19dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abaffd700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0abc102c90 0x7f0abc19dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abaffd700 1 -- 172.21.15.152:0/359840158 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc19e410 con 0x7f0abc1022f0 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abb7fe700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc1010f0 0x7f0abc1967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abbfff700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0abc102c90 0x7f0abc19dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.001+0000 7f0abaffd700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0abc1022f0 0x7f0abc196cf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000bb20 tx=0x7f0ab000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.002+0000 7f0ab8ff9700 1 -- 172.21.15.152:0/359840158 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0016070 con 0x7f0abc1022f0 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.002+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc06a890 con 0x7f0abc1022f0 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.002+0000 7f0ab8ff9700 1 -- 172.21.15.152:0/359840158 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0ab0009db0 con 0x7f0abc1022f0 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.002+0000 7f0ab8ff9700 1 -- 172.21.15.152:0/359840158 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab000ed20 con 0x7f0abc1022f0 2024-03-21T13:11:27.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.002+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc06ae20 con 0x7f0abc1022f0 2024-03-21T13:11:27.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.003+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc068e50 con 0x7f0abc1022f0 2024-03-21T13:11:27.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.004+0000 7f0ab8ff9700 1 -- 172.21.15.152:0/359840158 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 15) v1 ==== 92130+0+0 (secure 0 0 0) 0x7f0ab0019070 con 0x7f0abc1022f0 2024-03-21T13:11:27.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.004+0000 7f0ab8ff9700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0aa406e250 0x7f0aa4070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:27.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.004+0000 7f0ab8ff9700 1 -- 172.21.15.152:0/359840158 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f0ab008df90 con 0x7f0abc1022f0 2024-03-21T13:11:27.007 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.004+0000 7f0abb7fe700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0aa406e250 0x7f0aa4070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:27.007 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.005+0000 7f0abb7fe700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0aa406e250 0x7f0aa4070710 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0105c0 tx=0x7f0aac00b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:27.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.009+0000 7f0ab8ff9700 1 -- 172.21.15.152:0/359840158 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ab005b2d0 con 0x7f0abc1022f0 2024-03-21T13:11:27.228 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.227+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f0abc108450 con 0x7f0aa406e250 2024-03-21T13:11:27.228 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 podman[63382]: 2024-03-21 13:11:27.13439933 +0000 UTC m=+0.119027008 container create da50c6fe332a93fa2cbf1f71ba542f5a5768e117ab9e39110d96a19597a82802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-21T13:11:27.228 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 podman[63382]: 2024-03-21 13:11:27.037787398 +0000 UTC m=+0.022415084 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:11:27.235 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.234+0000 7f0ab8ff9700 1 -- 172.21.15.152:0/359840158 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f0abc108450 con 0x7f0aa406e250 2024-03-21T13:11:27.235 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled prometheus update... 2024-03-21T13:11:27.237 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0aa406e250 msgr2=0x7f0aa4070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:27.237 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0aa406e250 0x7f0aa4070710 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0105c0 tx=0x7f0aac00b4a0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.237 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0abc1022f0 msgr2=0x7f0abc196cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0abc1022f0 0x7f0abc196cf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000bb20 tx=0x7f0ab000be30 comp rx=0 tx=0).stop 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 shutdown_connections 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0abc102c90 0x7f0abc19dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f0aa406e250 0x7f0aa4070710 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0abc1010f0 0x7f0abc1967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 --2- 172.21.15.152:0/359840158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0abc1022f0 0x7f0abc196cf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 >> 172.21.15.152:0/359840158 conn(0x7f0abc0fc690 msgr2=0x7f0abc105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 shutdown_connections 2024-03-21T13:11:27.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:27.237+0000 7f0ac1ca6700 1 -- 172.21.15.152:0/359840158 wait complete. 2024-03-21T13:11:27.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 podman[63382]: 2024-03-21 13:11:27.337417537 +0000 UTC m=+0.322045220 container init da50c6fe332a93fa2cbf1f71ba542f5a5768e117ab9e39110d96a19597a82802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-21T13:11:27.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 podman[63382]: 2024-03-21 13:11:27.343365304 +0000 UTC m=+0.327992982 container start da50c6fe332a93fa2cbf1f71ba542f5a5768e117ab9e39110d96a19597a82802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-21T13:11:27.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 bash[63382]: da50c6fe332a93fa2cbf1f71ba542f5a5768e117ab9e39110d96a19597a82802 2024-03-21T13:11:27.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 systemd[1]: Started Ceph iscsi.iscsi.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:11:27.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug Started the configuration object watcher 2024-03-21T13:11:27.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug Checking for config object changes every 1s 2024-03-21T13:11:27.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug Processing osd blocklist entries for this node 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: pgmap v201: 132 pgs: 3 creating+activating, 4 creating+peering, 125 active+clean; 580 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 109 op/s 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/764502411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T13:11:28.261 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:28 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug Reading the configuration object to update local LIO configuration 2024-03-21T13:11:28.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug Configuration does not have an entry for this host(smithi152.front.sepia.ceph.com) - nothing to define to LIO 2024-03-21T13:11:28.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-21T13:11:28.285 DEBUG:teuthology.orchestra.run.smithi152:prometheus.a> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-21T13:11:28.287 INFO:tasks.cephadm:Adding node-exporter.a on smithi089 2024-03-21T13:11:28.288 INFO:tasks.cephadm:Adding node-exporter.b on smithi152 2024-03-21T13:11:28.288 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi089=a;smithi152=b' 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: pgmap v201: 132 pgs: 3 creating+activating, 4 creating+peering, 125 active+clean; 580 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 109 op/s 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/764502411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T13:11:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: pgmap v201: 132 pgs: 3 creating+activating, 4 creating+peering, 125 active+clean; 580 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 109 op/s 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/764502411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T13:11:28.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:28 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:28.570 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:29 smithi152 ceph-mon[34051]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:29 smithi152 ceph-mon[34051]: Saving service prometheus spec with placement smithi152=a;count:1 2024-03-21T13:11:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:29 smithi152 ceph-mon[34051]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T13:11:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:29 smithi089 ceph-mon[39209]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:29 smithi089 ceph-mon[39209]: Saving service prometheus spec with placement smithi152=a;count:1 2024-03-21T13:11:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:29 smithi089 ceph-mon[39209]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T13:11:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:29 smithi089 ceph-mon[30095]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:29 smithi089 ceph-mon[30095]: Saving service prometheus spec with placement smithi152=a;count:1 2024-03-21T13:11:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:29 smithi089 ceph-mon[30095]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T13:11:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:30 smithi152 ceph-mon[34051]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 5.0 KiB/s wr, 147 op/s 2024-03-21T13:11:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:30 smithi152 ceph-mon[34051]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T13:11:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:30 smithi089 ceph-mon[39209]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 5.0 KiB/s wr, 147 op/s 2024-03-21T13:11:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:30 smithi089 ceph-mon[39209]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T13:11:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:30 smithi089 ceph-mon[30095]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 5.0 KiB/s wr, 147 op/s 2024-03-21T13:11:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:30 smithi089 ceph-mon[30095]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T13:11:30.770 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled node-exporter update... 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 -- 172.21.15.152:0/3182390059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 msgr2=0x7f18201006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 --2- 172.21.15.152:0/3182390059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f18201006e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1814009990 tx=0x7f1814009ca0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 -- 172.21.15.152:0/3182390059 shutdown_connections 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 --2- 172.21.15.152:0/3182390059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820100c20 0x7f182010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 --2- 172.21.15.152:0/3182390059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820103800 0x7f1820103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 --2- 172.21.15.152:0/3182390059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f18201006e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 -- 172.21.15.152:0/3182390059 >> 172.21.15.152:0/3182390059 conn(0x7f18200fbe20 msgr2=0x7f18200fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:30.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 -- 172.21.15.152:0/3182390059 shutdown_connections 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.437+0000 7f18255ae700 1 -- 172.21.15.152:0/3182390059 wait complete. 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.438+0000 7f18255ae700 1 Processor -- start 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.438+0000 7f18255ae700 1 -- start start 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f1820197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820100c20 0x7f1820197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820103800 0x7f182019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182010d320 con 0x7f1820100c20 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182010d1a0 con 0x7f1820100280 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182010d4a0 con 0x7f1820103800 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f1820197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f1820197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:43896/0 (socket says 172.21.15.152:43896) 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 -- 172.21.15.152:0/1080352855 learned_addr learned my addr 172.21.15.152:0/1080352855 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:30.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181e7fc700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820100c20 0x7f1820197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181f7fe700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820103800 0x7f182019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 -- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820103800 msgr2=0x7f182019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820103800 0x7f182019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 -- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820100c20 msgr2=0x7f1820197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820100c20 0x7f1820197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 -- 172.21.15.152:0/1080352855 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182019f680 con 0x7f1820100280 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f181effd700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f1820197a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f181000b500 tx=0x7f181000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f1807fff700 1 -- 172.21.15.152:0/1080352855 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810015020 con 0x7f1820100280 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182019f970 con 0x7f1820100280 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f1807fff700 1 -- 172.21.15.152:0/1080352855 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1810011cb0 con 0x7f1820100280 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f1807fff700 1 -- 172.21.15.152:0/1080352855 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810010bd0 con 0x7f1820100280 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.439+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182019fee0 con 0x7f1820100280 2024-03-21T13:11:30.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.440+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1820068e50 con 0x7f1820100280 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.441+0000 7f1807fff700 1 -- 172.21.15.152:0/1080352855 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 92181+0+0 (secure 0 0 0) 0x7f1810018020 con 0x7f1820100280 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.441+0000 7f1807fff700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f180806e2e0 0x7f18080707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.442+0000 7f1807fff700 1 -- 172.21.15.152:0/1080352855 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f181008dc90 con 0x7f1820100280 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.442+0000 7f181e7fc700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f180806e2e0 0x7f18080707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.442+0000 7f181e7fc700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f180806e2e0 0x7f18080707a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f181400b5c0 tx=0x7f18140036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.447+0000 7f1807fff700 1 -- 172.21.15.152:0/1080352855 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f181001f020 con 0x7f1820100280 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.662+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi089=a;smithi152=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f18200625d0 con 0x7f180806e2e0 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.669+0000 7f1807fff700 1 -- 172.21.15.152:0/1080352855 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f18200625d0 con 0x7f180806e2e0 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f180806e2e0 msgr2=0x7f18080707a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f180806e2e0 0x7f18080707a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f181400b5c0 tx=0x7f18140036f0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 msgr2=0x7f1820197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f1820197a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f181000b500 tx=0x7f181000b810 comp rx=0 tx=0).stop 2024-03-21T13:11:30.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 shutdown_connections 2024-03-21T13:11:30.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820103800 0x7f182019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f180806e2e0 0x7f18080707a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100280 0x7f1820197a20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 --2- 172.21.15.152:0/1080352855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820100c20 0x7f1820197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:30.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 >> 172.21.15.152:0/1080352855 conn(0x7f18200fbe20 msgr2=0x7f18200fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:30.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 shutdown_connections 2024-03-21T13:11:30.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:30.672+0000 7f18255ae700 1 -- 172.21.15.152:0/1080352855 wait complete. 2024-03-21T13:11:31.530 DEBUG:teuthology.orchestra.run.smithi089:node-exporter.a> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-21T13:11:31.533 DEBUG:teuthology.orchestra.run.smithi152:node-exporter.b> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-21T13:11:31.536 INFO:tasks.cephadm:Adding alertmanager.a on smithi089 2024-03-21T13:11:31.536 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi089=a' 2024-03-21T13:11:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:31 smithi152 ceph-mon[34051]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi089=a;smithi152=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:31 smithi152 ceph-mon[34051]: Saving service node-exporter spec with placement smithi089=a;smithi152=b;count:2 2024-03-21T13:11:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:31 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:31 smithi152 ceph-mon[34051]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T13:11:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:31 smithi152 ceph-mon[34051]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-21T13:11:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[30095]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi089=a;smithi152=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[30095]: Saving service node-exporter spec with placement smithi089=a;smithi152=b;count:2 2024-03-21T13:11:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[30095]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T13:11:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[30095]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-21T13:11:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[39209]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi089=a;smithi152=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[39209]: Saving service node-exporter spec with placement smithi089=a;smithi152=b;count:2 2024-03-21T13:11:31.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:31.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[39209]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T13:11:31.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:31 smithi089 ceph-mon[39209]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-21T13:11:32.155 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:33 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:33 smithi152 ceph-mon[34051]: Deploying daemon prometheus.a on smithi152 2024-03-21T13:11:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:33 smithi152 ceph-mon[34051]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-03-21T13:11:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[39209]: Deploying daemon prometheus.a on smithi152 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[39209]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[30095]: Deploying daemon prometheus.a on smithi152 2024-03-21T13:11:33.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:33 smithi089 ceph-mon[30095]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-03-21T13:11:34.096 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 -- 172.21.15.152:0/2448996058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42701033c0 msgr2=0x7f4270103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:34.096 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 --2- 172.21.15.152:0/2448996058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42701033c0 0x7f4270103820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f42640099c0 tx=0x7f4264009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.096 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 -- 172.21.15.152:0/2448996058 shutdown_connections 2024-03-21T13:11:34.096 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 --2- 172.21.15.152:0/2448996058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4270103d60 0x7f427010a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.096 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 --2- 172.21.15.152:0/2448996058 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42701021c0 0x7f42701025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.096 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 --2- 172.21.15.152:0/2448996058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42701033c0 0x7f4270103820 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 -- 172.21.15.152:0/2448996058 >> 172.21.15.152:0/2448996058 conn(0x7f42700fd7a0 msgr2=0x7f42700ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 -- 172.21.15.152:0/2448996058 shutdown_connections 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 -- 172.21.15.152:0/2448996058 wait complete. 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 Processor -- start 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.094+0000 7f4276537700 1 -- start start 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4276537700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42701021c0 0x7f4270197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4276537700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42701033c0 0x7f4270197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4276537700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4270103d60 0x7f427019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4276537700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427010d2f0 con 0x7f42701021c0 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4276537700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427010d170 con 0x7f4270103d60 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4276537700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427010d470 con 0x7f42701033c0 2024-03-21T13:11:34.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4270103d60 0x7f427019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4270103d60 0x7f427019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:43916/0 (socket says 172.21.15.152:43916) 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 -- 172.21.15.152:0/317393231 learned_addr learned my addr 172.21.15.152:0/317393231 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f426ffff700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42701021c0 0x7f4270197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f426f7fe700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42701033c0 0x7f4270197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 -- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42701033c0 msgr2=0x7f4270197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42701033c0 0x7f4270197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 -- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42701021c0 msgr2=0x7f4270197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42701021c0 0x7f4270197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 -- 172.21.15.152:0/317393231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427019f5d0 con 0x7f4270103d60 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4274ad4700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4270103d60 0x7f427019ef30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4268007d70 tx=0x7f426800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f426d7fa700 1 -- 172.21.15.152:0/317393231 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268016020 con 0x7f4270103d60 2024-03-21T13:11:34.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f426d7fa700 1 -- 172.21.15.152:0/317393231 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f426800ac90 con 0x7f4270103d60 2024-03-21T13:11:34.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f426d7fa700 1 -- 172.21.15.152:0/317393231 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268011380 con 0x7f4270103d60 2024-03-21T13:11:34.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.095+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427019f8c0 con 0x7f4270103d60 2024-03-21T13:11:34.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.096+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427019fe80 con 0x7f4270103d60 2024-03-21T13:11:34.100 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.096+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4270068e50 con 0x7f4270103d60 2024-03-21T13:11:34.100 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.097+0000 7f426d7fa700 1 -- 172.21.15.152:0/317393231 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 92181+0+0 (secure 0 0 0) 0x7f4268019020 con 0x7f4270103d60 2024-03-21T13:11:34.100 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.097+0000 7f426d7fa700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f425806e2e0 0x7f42580707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:34.100 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.097+0000 7f426d7fa700 1 -- 172.21.15.152:0/317393231 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f426808e4a0 con 0x7f4270103d60 2024-03-21T13:11:34.100 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.097+0000 7f426ffff700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f425806e2e0 0x7f42580707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:34.100 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.098+0000 7f426ffff700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f425806e2e0 0x7f42580707a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4270103220 tx=0x7f4260009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:34.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.102+0000 7f426d7fa700 1 -- 172.21.15.152:0/317393231 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4268020020 con 0x7f4270103d60 2024-03-21T13:11:34.311 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.310+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi089=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f427002dd60 con 0x7f425806e2e0 2024-03-21T13:11:34.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.317+0000 7f426d7fa700 1 -- 172.21.15.152:0/317393231 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f427002dd60 con 0x7f425806e2e0 2024-03-21T13:11:34.318 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled alertmanager update... 2024-03-21T13:11:34.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.319+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f425806e2e0 msgr2=0x7f42580707a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:34.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.319+0000 7f4276537700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f425806e2e0 0x7f42580707a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4270103220 tx=0x7f4260009190 comp rx=0 tx=0).stop 2024-03-21T13:11:34.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.319+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4270103d60 msgr2=0x7f427019ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:34.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.319+0000 7f4276537700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4270103d60 0x7f427019ef30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4268007d70 tx=0x7f426800a340 comp rx=0 tx=0).stop 2024-03-21T13:11:34.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.319+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 shutdown_connections 2024-03-21T13:11:34.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.320+0000 7f4276537700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f425806e2e0 0x7f42580707a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.320+0000 7f4276537700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4270103d60 0x7f427019ef30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.320+0000 7f4276537700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42701021c0 0x7f4270197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.320+0000 7f4276537700 1 --2- 172.21.15.152:0/317393231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42701033c0 0x7f4270197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:34.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.320+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 >> 172.21.15.152:0/317393231 conn(0x7f42700fd7a0 msgr2=0x7f42700ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:34.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.320+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 shutdown_connections 2024-03-21T13:11:34.321 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:34.320+0000 7f4276537700 1 -- 172.21.15.152:0/317393231 wait complete. 2024-03-21T13:11:35.200 DEBUG:teuthology.orchestra.run.smithi089:alertmanager.a> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-21T13:11:35.203 INFO:tasks.cephadm:Adding grafana.a on smithi152 2024-03-21T13:11:35.203 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi152=a' 2024-03-21T13:11:35.446 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:35.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:35 smithi152 ceph-mon[34051]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi089=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:35.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:35 smithi152 ceph-mon[34051]: Saving service alertmanager spec with placement smithi089=a;count:1 2024-03-21T13:11:35.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:35 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:35.568 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:35 smithi152 ceph-mon[34051]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 3.0 KiB/s wr, 91 op/s 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[39209]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi089=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[39209]: Saving service alertmanager spec with placement smithi089=a;count:1 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[39209]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 3.0 KiB/s wr, 91 op/s 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[30095]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi089=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[30095]: Saving service alertmanager spec with placement smithi089=a;count:1 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:35 smithi089 ceph-mon[30095]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 3.0 KiB/s wr, 91 op/s 2024-03-21T13:11:37.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:11:38.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.059+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/2662181257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01010b0 msgr2=0x7ff1b01014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.059+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/2662181257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01010b0 0x7ff1b01014d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40099c0 tx=0x7ff1a4009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/2662181257 shutdown_connections 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/2662181257 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1b0102c50 0x7ff1b0109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/2662181257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01010b0 0x7ff1b01014d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/2662181257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1b01022b0 0x7ff1b0102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/2662181257 >> 172.21.15.152:0/2662181257 conn(0x7ff1b00fc650 msgr2=0x7ff1b00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/2662181257 shutdown_connections 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/2662181257 wait complete. 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 Processor -- start 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 -- start start 2024-03-21T13:11:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1b01010b0 0x7ff1b0199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.060+0000 7ff1b6dc6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01022b0 0x7ff1b019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b6dc6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1b0102c50 0x7ff1b01a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b6dc6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b010c1a0 con 0x7ff1b0102c50 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b6dc6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b010c020 con 0x7ff1b01022b0 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b6dc6700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b010c320 con 0x7ff1b01010b0 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01022b0 0x7ff1b019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01022b0 0x7ff1b019a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:37314/0 (socket says 172.21.15.152:37314) 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 -- 172.21.15.152:0/4135737637 learned_addr learned my addr 172.21.15.152:0/4135737637 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b4b62700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1b01010b0 0x7ff1b0199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b5363700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1b0102c50 0x7ff1b01a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 -- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1b01010b0 msgr2=0x7ff1b0199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1b01010b0 0x7ff1b0199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 -- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1b0102c50 msgr2=0x7ff1b01a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1b0102c50 0x7ff1b01a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 -- 172.21.15.152:0/4135737637 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b01a1780 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b4b62700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1b01010b0 0x7ff1b0199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b5363700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1b0102c50 0x7ff1b01a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1affff700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01022b0 0x7ff1b019a060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00e9a0 tx=0x7ff19c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1adffb700 1 -- 172.21.15.152:0/4135737637 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c00cbd0 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1adffb700 1 -- 172.21.15.152:0/4135737637 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff19c00cd70 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b01a1a70 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1adffb700 1 -- 172.21.15.152:0/4135737637 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c018680 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.061+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b01a1fb0 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.062+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b0068e50 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.063+0000 7ff1adffb700 1 -- 172.21.15.152:0/4135737637 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 92181+0+0 (secure 0 0 0) 0x7ff19c010430 con 0x7ff1b01022b0 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.063+0000 7ff1adffb700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1a006e2e0 0x7ff1a00707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:38.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.063+0000 7ff1adffb700 1 -- 172.21.15.152:0/4135737637 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7148+0+0 (secure 0 0 0) 0x7ff19c08e9d0 con 0x7ff1b01022b0 2024-03-21T13:11:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.063+0000 7ff1b4b62700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1a006e2e0 0x7ff1a00707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.064+0000 7ff1b4b62700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1a006e2e0 0x7ff1a00707a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4000c00 tx=0x7ff1a4003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.068+0000 7ff1adffb700 1 -- 172.21.15.152:0/4135737637 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff19c05bcb0 con 0x7ff1b01022b0 2024-03-21T13:11:38.278 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.277+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 --> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1b00fe8a0 con 0x7ff1a006e2e0 2024-03-21T13:11:38.278 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:38 smithi152 ceph-mon[34051]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.5 KiB/s wr, 48 op/s 2024-03-21T13:11:38.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:38 smithi089 ceph-mon[30095]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.5 KiB/s wr, 48 op/s 2024-03-21T13:11:38.279 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:38 smithi089 ceph-mon[39209]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.5 KiB/s wr, 48 op/s 2024-03-21T13:11:38.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.285+0000 7ff1adffb700 1 -- 172.21.15.152:0/4135737637 <== mgr.14152 v2:172.21.15.89:6800/4277445380 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff1b00fe8a0 con 0x7ff1a006e2e0 2024-03-21T13:11:38.286 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled grafana update... 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1a006e2e0 msgr2=0x7ff1a00707a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1a006e2e0 0x7ff1a00707a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4000c00 tx=0x7ff1a4003680 comp rx=0 tx=0).stop 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01022b0 msgr2=0x7ff1b019a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01022b0 0x7ff1b019a060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00e9a0 tx=0x7ff19c00ed60 comp rx=0 tx=0).stop 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 shutdown_connections 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1b0102c50 0x7ff1b01a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1b01010b0 0x7ff1b0199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1a006e2e0 0x7ff1a00707a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 --2- 172.21.15.152:0/4135737637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01022b0 0x7ff1b019a060 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 >> 172.21.15.152:0/4135737637 conn(0x7ff1b00fc650 msgr2=0x7ff1b00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:38.289 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 shutdown_connections 2024-03-21T13:11:38.289 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:38.287+0000 7ff1b6dc6700 1 -- 172.21.15.152:0/4135737637 wait complete. 2024-03-21T13:11:39.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:39.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:39 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:39.299 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:39.299 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:39 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:39 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:40.053 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:40 smithi152 ceph-mon[34051]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:40.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:40 smithi152 ceph-mon[34051]: Saving service grafana spec with placement smithi152=a;count:1 2024-03-21T13:11:40.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:40 smithi152 ceph-mon[34051]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-21T13:11:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:40 smithi089 ceph-mon[30095]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:40 smithi089 ceph-mon[30095]: Saving service grafana spec with placement smithi152=a;count:1 2024-03-21T13:11:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:40 smithi089 ceph-mon[30095]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-21T13:11:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:40 smithi089 ceph-mon[39209]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi152=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:11:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:40 smithi089 ceph-mon[39209]: Saving service grafana spec with placement smithi152=a;count:1 2024-03-21T13:11:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:40 smithi089 ceph-mon[39209]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-21T13:11:40.390 DEBUG:teuthology.orchestra.run.smithi152:grafana.a> sudo journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@grafana.a.service 2024-03-21T13:11:40.392 INFO:tasks.cephadm:Setting up client nodes... 2024-03-21T13:11:40.393 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T13:11:40.630 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:42.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 -- 172.21.15.89:0/3565248055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801022e0 msgr2=0x7f1f80102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:42.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/3565248055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801022e0 0x7f1f80102740 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0099c0 tx=0x7f1f6c009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 -- 172.21.15.89:0/3565248055 shutdown_connections 2024-03-21T13:11:42.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/3565248055 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f80102c80 0x7f1f80109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/3565248055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f801010e0 0x7f1f80101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/3565248055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801022e0 0x7f1f80102740 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 -- 172.21.15.89:0/3565248055 >> 172.21.15.89:0/3565248055 conn(0x7f1f800fc680 msgr2=0x7f1f800feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:42.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 -- 172.21.15.89:0/3565248055 shutdown_connections 2024-03-21T13:11:42.289 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:42 smithi089 ceph-mon[30095]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-21T13:11:42.289 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:42 smithi089 ceph-mon[39209]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-21T13:11:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.287+0000 7f1f86b2a700 1 -- 172.21.15.89:0/3565248055 wait complete. 2024-03-21T13:11:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 Processor -- start 2024-03-21T13:11:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 -- start start 2024-03-21T13:11:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801010e0 0x7f1f8006a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f801022e0 0x7f1f8019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f80102c80 0x7f1f8019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8010c210 con 0x7f1f801010e0 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8010c090 con 0x7f1f80102c80 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f86b2a700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8010c390 con 0x7f1f801022e0 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f7ffff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f801022e0 0x7f1f8019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f7ffff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f801022e0 0x7f1f8019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:58664/0 (socket says 172.21.15.89:58664) 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f7ffff700 1 -- 172.21.15.89:0/4277426655 learned_addr learned my addr 172.21.15.89:0/4277426655 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f848c6700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801010e0 0x7f1f8006a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.288+0000 7f1f850c7700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f80102c80 0x7f1f8019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7ffff700 1 -- 172.21.15.89:0/4277426655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f80102c80 msgr2=0x7f1f8019ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7ffff700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f80102c80 0x7f1f8019ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7ffff700 1 -- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801010e0 msgr2=0x7f1f8006a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7ffff700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801010e0 0x7f1f8006a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7ffff700 1 -- 172.21.15.89:0/4277426655 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8019f560 con 0x7f1f801022e0 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f850c7700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f80102c80 0x7f1f8019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7ffff700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f801022e0 0x7f1f8019c650 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c006010 tx=0x7f1f6c0055f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7dffb700 1 -- 172.21.15.89:0/4277426655 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c026070 con 0x7f1f801022e0 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7dffb700 1 -- 172.21.15.89:0/4277426655 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1f6c003e50 con 0x7f1f801022e0 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8019f7f0 con 0x7f1f801022e0 2024-03-21T13:11:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f7dffb700 1 -- 172.21.15.89:0/4277426655 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00fbe0 con 0x7f1f801022e0 2024-03-21T13:11:42.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.289+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8019fd20 con 0x7f1f801022e0 2024-03-21T13:11:42.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.290+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f800506a0 con 0x7f1f801022e0 2024-03-21T13:11:42.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.291+0000 7f1f7dffb700 1 -- 172.21.15.89:0/4277426655 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 16) v1 ==== 92181+0+0 (secure 0 0 0) 0x7f1f6c029070 con 0x7f1f801022e0 2024-03-21T13:11:42.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.291+0000 7f1f7dffb700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f1f7006e390 0x7f1f70070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:42.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.291+0000 7f1f7dffb700 1 -- 172.21.15.89:0/4277426655 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f1f6c08f160 con 0x7f1f801022e0 2024-03-21T13:11:42.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.291+0000 7f1f848c6700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f1f7006e390 0x7f1f70070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:42.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.292+0000 7f1f848c6700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f1f7006e390 0x7f1f70070850 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1f80102140 tx=0x7f1f74009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:42.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.296+0000 7f1f7dffb700 1 -- 172.21.15.89:0/4277426655 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1f6c013070 con 0x7f1f801022e0 2024-03-21T13:11:42.313 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:42 smithi152 ceph-mon[34051]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-21T13:11:42.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.575+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1f8002da30 con 0x7f1f801022e0 2024-03-21T13:11:42.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.579+0000 7f1f7dffb700 1 -- 172.21.15.89:0/4277426655 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1f6c05c4c0 con 0x7f1f801022e0 2024-03-21T13:11:42.580 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-03-21T13:11:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOMvxlTf1jIhAAFbWX6aN/QGuzsjzlLVGF4Q== 2024-03-21T13:11:42.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.581+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f1f7006e390 msgr2=0x7f1f70070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:42.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.581+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f1f7006e390 0x7f1f70070850 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1f80102140 tx=0x7f1f74009190 comp rx=0 tx=0).stop 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.581+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f801022e0 msgr2=0x7f1f8019c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.581+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f801022e0 0x7f1f8019c650 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c006010 tx=0x7f1f6c0055f0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 shutdown_connections 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7f1f7006e390 0x7f1f70070850 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f80102c80 0x7f1f8019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f801010e0 0x7f1f8006a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 --2- 172.21.15.89:0/4277426655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f801022e0 0x7f1f8019c650 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 >> 172.21.15.89:0/4277426655 conn(0x7f1f800fc680 msgr2=0x7f1f800fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 shutdown_connections 2024-03-21T13:11:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:42.582+0000 7f1f86b2a700 1 -- 172.21.15.89:0/4277426655 wait complete. 2024-03-21T13:11:43.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4277426655' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:43.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:43.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T13:11:43.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4277426655' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-21T13:11:43.158 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:43 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-21T13:11:43.181 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:11:43.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-21T13:11:43.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-21T13:11:43.224 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T13:11:43.253 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4277426655' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:43.254 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:43.254 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T13:11:43.254 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-21T13:11:43.254 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-21T13:11:43.254 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:43 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-21T13:11:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:44 smithi152 ceph-mon[34051]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-21T13:11:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:44 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-21T13:11:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:44 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-21T13:11:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:44 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-21T13:11:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:44 smithi152 ceph-mon[34051]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[30095]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[30095]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[39209]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-21T13:11:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:44 smithi089 ceph-mon[39209]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T13:11:44.739 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:11:45.960 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:45 smithi152 ceph-mon[34051]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T13:11:45.960 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:45 smithi152 ceph-mon[34051]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:11:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:45 smithi089 ceph-mon[30095]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T13:11:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:45 smithi089 ceph-mon[30095]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:11:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:45 smithi089 ceph-mon[39209]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T13:11:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:45 smithi089 ceph-mon[39209]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:11:46.321 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 podman[65256]: 2024-03-21 13:11:46.186535959 +0000 UTC m=+0.607543617 container create 77b19770aaad14118de33ecefbe1ece1f535beb863a97a78a7b9e41490e6b487 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:11:46.776 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 podman[65256]: 2024-03-21 13:11:46.455630663 +0000 UTC m=+0.876638324 container init 77b19770aaad14118de33ecefbe1ece1f535beb863a97a78a7b9e41490e6b487 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:11:46.776 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 podman[65256]: 2024-03-21 13:11:46.461041094 +0000 UTC m=+0.882048745 container start 77b19770aaad14118de33ecefbe1ece1f535beb863a97a78a7b9e41490e6b487 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.484Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.485Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.485Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi152 (none))" 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.485Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.485Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.487Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.487Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.488Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.488Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.490Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.490Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.04µs 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.490Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T13:11:46.777 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.490Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.490Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.599µs wal_replay_duration=210.907µs wbl_replay_duration=94ns total_replay_duration=250.579µs 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.491Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.491Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.491Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.511Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.76299ms db_storage=635ns remote_storage=1.338µs web_handler=329ns query_engine=537ns scrape=454.408µs scrape_sd=69.752µs notify=680ns notify_sd=1.392µs rules=19.989778ms tracing=8.304µs 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.511Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:11:46.512Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 bash[65256]: 77b19770aaad14118de33ecefbe1ece1f535beb863a97a78a7b9e41490e6b487 2024-03-21T13:11:46.778 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:11:46 smithi152 systemd[1]: Started Ceph prometheus.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 -- 172.21.15.152:0/2377127090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 msgr2=0x7fa550102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 --2- 172.21.15.152:0/2377127090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550102610 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0099c0 tx=0x7fa54c009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 -- 172.21.15.152:0/2377127090 shutdown_connections 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 --2- 172.21.15.152:0/2377127090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa550103d90 0x7fa55010a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 --2- 172.21.15.152:0/2377127090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550102610 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 --2- 172.21.15.152:0/2377127090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5501033f0 0x7fa550103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 -- 172.21.15.152:0/2377127090 >> 172.21.15.152:0/2377127090 conn(0x7fa5500fd770 msgr2=0x7fa5500ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 -- 172.21.15.152:0/2377127090 shutdown_connections 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.037+0000 7fa5572ee700 1 -- 172.21.15.152:0/2377127090 wait complete. 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 Processor -- start 2024-03-21T13:11:47.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 -- start start 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5501033f0 0x7fa550074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa550103d90 0x7fa550074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55010d2e0 con 0x7fa5501033f0 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55010d160 con 0x7fa5501021f0 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa5572ee700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55010d460 con 0x7fa550103d90 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55508a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55508a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:58830/0 (socket says 172.21.15.152:58830) 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55508a700 1 -- 172.21.15.152:0/3014877883 learned_addr learned my addr 172.21.15.152:0/3014877883 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa554889700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5501033f0 0x7fa550074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55588b700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa550103d90 0x7fa550074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55508a700 1 -- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa550103d90 msgr2=0x7fa550074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:47.920 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55508a700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa550103d90 0x7fa550074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55508a700 1 -- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5501033f0 msgr2=0x7fa550074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.038+0000 7fa55508a700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5501033f0 0x7fa550074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.039+0000 7fa55508a700 1 -- 172.21.15.152:0/3014877883 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa550075370 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.039+0000 7fa55508a700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550075f30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006010 tx=0x7fa54c0055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.039+0000 7fa5467fc700 1 -- 172.21.15.152:0/3014877883 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c026070 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.039+0000 7fa5467fc700 1 -- 172.21.15.152:0/3014877883 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa54c003e50 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.039+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5501a19a0 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.039+0000 7fa5467fc700 1 -- 172.21.15.152:0/3014877883 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c00fc10 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.039+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5501a1df0 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.040+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5500506a0 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.040+0000 7fa5467fc700 1 -- 172.21.15.152:0/3014877883 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 92181+0+0 (secure 0 0 0) 0x7fa54c029070 con 0x7fa5501021f0 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.041+0000 7fa5467fc700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fa53c06e390 0x7fa53c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:47.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.041+0000 7fa5467fc700 1 -- 172.21.15.152:0/3014877883 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7207+0+0 (secure 0 0 0) 0x7fa54c08f0e0 con 0x7fa5501021f0 2024-03-21T13:11:47.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.041+0000 7fa554889700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fa53c06e390 0x7fa53c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:47.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.041+0000 7fa554889700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fa53c06e390 0x7fa53c070850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa540009c60 tx=0x7fa540009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:47.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.045+0000 7fa5467fc700 1 -- 172.21.15.152:0/3014877883 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa54c013070 con 0x7fa5501021f0 2024-03-21T13:11:47.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.319+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa55002da30 con 0x7fa5501021f0 2024-03-21T13:11:47.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.326+0000 7fa5467fc700 1 -- 172.21.15.152:0/3014877883 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa54c05c400 con 0x7fa5501021f0 2024-03-21T13:11:47.922 INFO:teuthology.orchestra.run.smithi152.stdout:[client.1] 2024-03-21T13:11:47.922 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQATMvxlvKwnExAA5sWhgguaRcgKtmAsMhcfVg== 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fa53c06e390 msgr2=0x7fa53c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fa53c06e390 0x7fa53c070850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa540009c60 tx=0x7fa540009260 comp rx=0 tx=0).stop 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 msgr2=0x7fa550075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550075f30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006010 tx=0x7fa54c0055f0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 shutdown_connections 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa550103d90 0x7fa550074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fa53c06e390 0x7fa53c070850 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5501021f0 0x7fa550075f30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 --2- 172.21.15.152:0/3014877883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5501033f0 0x7fa550074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 >> 172.21.15.152:0/3014877883 conn(0x7fa5500fd770 msgr2=0x7fa5501066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.328+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 shutdown_connections 2024-03-21T13:11:47.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-21T13:11:47.329+0000 7fa5572ee700 1 -- 172.21.15.152:0/3014877883 wait complete. 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.152:0/3014877883' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:47.924 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T13:11:47.925 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:11:47.925 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: ignoring --setuser ceph since I am not root 2024-03-21T13:11:47.925 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: ignoring --setgroup ceph since I am not root 2024-03-21T13:11:47.925 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mgr[36154]: -- 172.21.15.152:0/2605655730 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5649b39d24e0 con 0x5649b39a8c00 2024-03-21T13:11:47.925 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:47 smithi152 ceph-mgr[36154]: -- 172.21.15.152:0/2605655730 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5649b39d45a0 con 0x5649b39a8c00 2024-03-21T13:11:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T13:11:48.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T13:11:48.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.152:0/3014877883' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:48.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:48.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T13:11:48.076 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ignoring --setuser ceph since I am not root 2024-03-21T13:11:48.076 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ignoring --setgroup ceph since I am not root 2024-03-21T13:11:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:48.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:48.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' 2024-03-21T13:11:48.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T13:11:48.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T13:11:48.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.152:0/3014877883' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:48.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T13:11:48.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T13:11:48.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:48.219+0000 7ff7ac254200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:11:48.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:48.315+0000 7ff7ac254200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:11:48.574 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:48 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:48.220+0000 7f74a6917200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:11:48.575 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:48 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:48.315+0000 7f74a6917200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:11:48.949 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:48 smithi089 ceph-mon[39209]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T13:11:48.949 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:48 smithi089 ceph-mon[39209]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T13:11:48.949 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:48 smithi089 ceph-mon[30095]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T13:11:48.949 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:48 smithi089 ceph-mon[30095]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T13:11:48.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:48 smithi152 ceph-mon[34051]: from='mgr.14152 172.21.15.89:0/3401010717' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T13:11:48.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:48 smithi152 ceph-mon[34051]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T13:11:49.276 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:48 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:48.948+0000 7f74a6917200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:11:49.276 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.039+0000 7f74a6917200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:11:49.276 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.180+0000 7f74a6917200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:11:49.276 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.275+0000 7f74a6917200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:11:49.296 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:48.958+0000 7ff7ac254200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:11:49.296 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.053+0000 7ff7ac254200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:11:49.296 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.198+0000 7ff7ac254200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:11:49.296 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.295+0000 7ff7ac254200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:11:49.414 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-21T13:11:49.414 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-21T13:11:49.414 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-21T13:11:49.499 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-21T13:11:49.499 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-21T13:11:49.499 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-21T13:11:49.527 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.358+0000 7f74a6917200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:11:49.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.379+0000 7ff7ac254200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:11:49.762 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:49.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.626+0000 7f74a6917200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:11:49.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.762+0000 7f74a6917200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:11:49.947 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.653+0000 7ff7ac254200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:11:49.947 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.764+0000 7ff7ac254200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:11:49.947 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.850+0000 7ff7ac254200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:11:49.947 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:49.946+0000 7ff7ac254200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:11:50.144 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.853+0000 7f74a6917200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:11:50.144 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:49 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:49.952+0000 7f74a6917200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:11:50.145 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:50 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:50.143+0000 7f74a6917200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:11:50.321 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:50 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:50.124+0000 7ff7ac254200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 -- 172.21.15.89:0/3446574803 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee806a930 msgr2=0x7fcee806ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 --2- 172.21.15.89:0/3446574803 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee806a930 0x7fcee806ad10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0099c0 tx=0x7fcedc009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 -- 172.21.15.89:0/3446574803 shutdown_connections 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 --2- 172.21.15.89:0/3446574803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee8107510 0x7fcee810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 --2- 172.21.15.89:0/3446574803 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee806a930 0x7fcee806ad10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 --2- 172.21.15.89:0/3446574803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcee806b2e0 0x7fcee8106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 -- 172.21.15.89:0/3446574803 >> 172.21.15.89:0/3446574803 conn(0x7fcee80770e0 msgr2=0x7fcee80774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 -- 172.21.15.89:0/3446574803 shutdown_connections 2024-03-21T13:11:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.380+0000 7fceee024700 1 -- 172.21.15.89:0/3446574803 wait complete. 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 Processor -- start 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 -- start start 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcee806a930 0x7fcee8197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee806b2e0 0x7fcee81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee8107510 0x7fcee819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8111910 con 0x7fcee806a930 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8111790 con 0x7fcee806b2e0 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fceee024700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8111a90 con 0x7fcee8107510 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fcee7fff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee8107510 0x7fcee819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fcee7fff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee8107510 0x7fcee819c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:46810/0 (socket says 172.21.15.89:46810) 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fcee77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcee806a930 0x7fcee8197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fcee7fff700 1 -- 172.21.15.89:0/4153329919 learned_addr learned my addr 172.21.15.89:0/4153329919 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:11:51.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.381+0000 7fcee6ffd700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee806b2e0 0x7fcee81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:51.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee77fe700 1 -- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee8107510 msgr2=0x7fcee819c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:51.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee77fe700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee8107510 0x7fcee819c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee77fe700 1 -- 172.21.15.89:0/4153329919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee806b2e0 msgr2=0x7fcee81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:51.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee77fe700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee806b2e0 0x7fcee81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee77fe700 1 -- 172.21.15.89:0/4153329919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee819ce90 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee7fff700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee8107510 0x7fcee819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee77fe700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcee806a930 0x7fcee8197e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0098e0 tx=0x7fcedc0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee4ff9700 1 -- 172.21.15.89:0/4153329919 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc026070 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee819d120 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fcee4ff9700 1 -- 172.21.15.89:0/4153329919 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcedc003e50 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.382+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee81a4a00 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.383+0000 7fcee4ff9700 1 -- 172.21.15.89:0/4153329919 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc00f940 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.383+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee8068e50 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.383+0000 7fcee4ff9700 1 -- 172.21.15.89:0/4153329919 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 92195+0+0 (secure 0 0 0) 0x7fcedc029070 con 0x7fcee806a930 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.384+0000 7fcee4ff9700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 0x7fced00707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.384+0000 7fcee6ffd700 1 -- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 msgr2=0x7fced00707f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4277445380 2024-03-21T13:11:51.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.384+0000 7fcee4ff9700 1 -- 172.21.15.89:0/4153329919 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7207+0+0 (secure 0 0 0) 0x7fcedc08eff0 con 0x7fcee806a930 2024-03-21T13:11:51.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.384+0000 7fcee6ffd700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 0x7fced00707f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:11:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.393+0000 7fcee4ff9700 1 -- 172.21.15.89:0/4153329919 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcedc013070 con 0x7fcee806a930 2024-03-21T13:11:51.394 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:51.175+0000 7f74a6917200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:11:51.394 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:51.311+0000 7f74a6917200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:11:51.553 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:51 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:51.088+0000 7ff7ac254200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:11:51.553 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:51 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:51.218+0000 7ff7ac254200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:11:51.553 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:51 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:51.552+0000 7ff7ac254200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:11:51.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.585+0000 7fcee6ffd700 1 -- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 msgr2=0x7fced00707f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4277445380 2024-03-21T13:11:51.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.585+0000 7fcee6ffd700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 0x7fced00707f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T13:11:51.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.655+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fcee81a4bd0 con 0x7fcee806a930 2024-03-21T13:11:51.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.660+0000 7fcee4ff9700 1 -- 172.21.15.89:0/4153329919 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+176246 (secure 0 0 0) 0x7fcedc05c290 con 0x7fcee806a930 2024-03-21T13:11:51.663 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:11:51.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 msgr2=0x7fced00707f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:11:51.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 0x7fced00707f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcee806a930 msgr2=0x7fcee8197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:51.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcee806a930 0x7fcee8197e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0098e0 tx=0x7fcedc0045e0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 shutdown_connections 2024-03-21T13:11:51.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcee8107510 0x7fcee819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcee806a930 0x7fcee8197e80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7fced006e330 0x7fced00707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 --2- 172.21.15.89:0/4153329919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee806b2e0 0x7fcee81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:51.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 >> 172.21.15.89:0/4153329919 conn(0x7fcee80770e0 msgr2=0x7fcee80fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:51.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 shutdown_connections 2024-03-21T13:11:51.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:51.664+0000 7fceee024700 1 -- 172.21.15.89:0/4153329919 wait complete. 2024-03-21T13:11:51.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4153329919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T13:11:51.821 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:51 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:51.642+0000 7ff7ac254200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:11:51.963 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4153329919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T13:11:51.963 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:51.665+0000 7f74a6917200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:11:51.963 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:51.762+0000 7f74a6917200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:11:51.963 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4153329919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T13:11:52.316 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:51 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:51.839+0000 7ff7ac254200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:11:52.316 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:52 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:52.026+0000 7ff7ac254200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:11:52.324 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:51.962+0000 7f74a6917200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:11:52.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:52.152+0000 7f74a6917200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:11:52.357 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":4277445380},{"type":"v1","addr":"172.21.15.89:6801","nonce":4277445380}]},"active_addr":"172.21.15.89:6801/4277445380","active_change":"2024-03-21T13:06:42.905138+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.89:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":319612939}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":391889855}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":197165808}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":4124817476}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":26756681}]}]} 2024-03-21T13:11:52.361 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-21T13:11:52.362 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T13:11:52.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T13:11:52.571 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:52 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:52.315+0000 7ff7ac254200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:11:52.647 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:52.726 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:52.450+0000 7f74a6917200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:11:53.033 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:52.725+0000 7f74a6917200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:11:53.033 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:52.972+0000 7f74a6917200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:11:53.071 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:52 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:52.577+0000 7ff7ac254200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:11:53.071 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:52 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:52.815+0000 7ff7ac254200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:11:54.071 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:53 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:53.616+0000 7ff7ac254200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:11:54.074 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:53 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:53.798+0000 7f74a6917200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:11:54.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 -- 172.21.15.89:0/491430449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 msgr2=0x7ff1cc106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:54.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/491430449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 0x7ff1cc106dd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0099c0 tx=0x7ff1bc009cd0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 -- 172.21.15.89:0/491430449 shutdown_connections 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/491430449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1cc1074b0 0x7ff1cc10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/491430449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 0x7ff1cc1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/491430449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 0x7ff1cc106dd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 -- 172.21.15.89:0/491430449 >> 172.21.15.89:0/491430449 conn(0x7ff1cc0fdc40 msgr2=0x7ff1cc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 -- 172.21.15.89:0/491430449 shutdown_connections 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.320+0000 7ff1d40d9700 1 -- 172.21.15.89:0/491430449 wait complete. 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 Processor -- start 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 -- start start 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 0x7ff1cc074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 0x7ff1cc0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:54.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1cc1074b0 0x7ff1cc074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc111920 con 0x7ff1cc102d80 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1117a0 con 0x7ff1cc1074b0 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d40d9700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc111aa0 con 0x7ff1cc1023d0 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1e75700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 0x7ff1cc074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1674700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 0x7ff1cc0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1e75700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 0x7ff1cc074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:46834/0 (socket says 172.21.15.89:46834) 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1e75700 1 -- 172.21.15.89:0/4289576095 learned_addr learned my addr 172.21.15.89:0/4289576095 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d2676700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1cc1074b0 0x7ff1cc074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1e75700 1 -- 172.21.15.89:0/4289576095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1cc1074b0 msgr2=0x7ff1cc074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1e75700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1cc1074b0 0x7ff1cc074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1e75700 1 -- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 msgr2=0x7ff1cc0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.321+0000 7ff1d1e75700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 0x7ff1cc0749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.322+0000 7ff1d1e75700 1 -- 172.21.15.89:0/4289576095 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc1a63b0 con 0x7ff1cc1023d0 2024-03-21T13:11:54.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.322+0000 7ff1d1e75700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 0x7ff1cc074490 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800eb30 tx=0x7ff1c800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:54.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.322+0000 7ff1c2ffd700 1 -- 172.21.15.89:0/4289576095 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800cbd0 con 0x7ff1cc1023d0 2024-03-21T13:11:54.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.322+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc1a66a0 con 0x7ff1cc1023d0 2024-03-21T13:11:54.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.322+0000 7ff1c2ffd700 1 -- 172.21.15.89:0/4289576095 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff1c80044a0 con 0x7ff1cc1023d0 2024-03-21T13:11:54.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.322+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1a6c00 con 0x7ff1cc1023d0 2024-03-21T13:11:54.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.323+0000 7ff1c2ffd700 1 -- 172.21.15.89:0/4289576095 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8003d40 con 0x7ff1cc1023d0 2024-03-21T13:11:54.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.323+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc068e50 con 0x7ff1cc1023d0 2024-03-21T13:11:54.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.324+0000 7ff1c2ffd700 1 -- 172.21.15.89:0/4289576095 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 17) v1 ==== 92195+0+0 (secure 0 0 0) 0x7ff1c8003ee0 con 0x7ff1cc1023d0 2024-03-21T13:11:54.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.324+0000 7ff1c2ffd700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 0x7ff1b80707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:54.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.325+0000 7ff1d1674700 1 -- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 msgr2=0x7ff1b80707f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4277445380 2024-03-21T13:11:54.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.325+0000 7ff1d1674700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 0x7ff1b80707f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T13:11:54.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.325+0000 7ff1c2ffd700 1 -- 172.21.15.89:0/4289576095 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7207+0+0 (secure 0 0 0) 0x7ff1c808ef10 con 0x7ff1cc1023d0 2024-03-21T13:11:54.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.329+0000 7ff1c2ffd700 1 -- 172.21.15.89:0/4289576095 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff1c805c230 con 0x7ff1cc1023d0 2024-03-21T13:11:54.513 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:54.190+0000 7ff7ac254200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:11:54.514 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:54.290+0000 7ff7ac254200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:11:54.514 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:54.513+0000 7ff7ac254200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:11:54.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.526+0000 7ff1d1674700 1 -- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 msgr2=0x7ff1b80707f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4277445380 2024-03-21T13:11:54.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.526+0000 7ff1d1674700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 0x7ff1b80707f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T13:11:54.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.531+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1cc1a6ef0 con 0x7ff1cc1023d0 2024-03-21T13:11:54.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.533+0000 7ff1c2ffd700 1 -- 172.21.15.89:0/4289576095 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+22726 (secure 0 0 0) 0x7ff1c805bb30 con 0x7ff1cc1023d0 2024-03-21T13:11:54.534 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:11:54.534 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"fsid":"9c355834-e783-11ee-95cd-87774f69a715","created":"2024-03-21T13:05:40.974140+0000","modified":"2024-03-21T13:11:44.583114+0000","last_up_change":"2024-03-21T13:11:02.379434+0000","last_in_change":"2024-03-21T13:10:44.727962+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T13:09:01.242029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T13:11:14.467225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T13:11:15.118984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T13:11:16.456191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T13:11:18.540730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T13:11:20.611141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4258f99e-44cf-4cd0-a093-00e1192a8ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6803","nonce":1699462714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6805","nonce":1699462714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6809","nonce":1699462714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6807","nonce":1699462714}]},"public_addr":"172.21.15.89:6803/1699462714","cluster_addr":"172.21.15.89:6805/1699462714","heartbeat_back_addr":"172.21.15.89:6809/1699462714","heartbeat_front_addr":"172.21.15.89:6807/1699462714","state":["exists","up"]},{"osd":1,"uuid":"debf8a73-c369-49ed-8886-57f2ca61e6d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6811","nonce":3822115704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6813","nonce":3822115704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6817","nonce":3822115704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6815","nonce":3822115704}]},"public_addr":"172.21.15.89:6811/3822115704","cluster_addr":"172.21.15.89:6813/3822115704","heartbeat_back_addr":"172.21.15.89:6817/3822115704","heartbeat_front_addr":"172.21.15.89:6815/3822115704","state":["exists","up"]},{"osd":2,"uuid":"2686770f-bbbc-4082-bb00-9f6f10f8783b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6819","nonce":2934798428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6821","nonce":2934798428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6825","nonce":2934798428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6823","nonce":2934798428}]},"public_addr":"172.21.15.89:6819/2934798428","cluster_addr":"172.21.15.89:6821/2934798428","heartbeat_back_addr":"172.21.15.89:6825/2934798428","heartbeat_front_addr":"172.21.15.89:6823/2934798428","state":["exists","up"]},{"osd":3,"uuid":"4c996867-ce38-4ae7-b934-5faf2ce2fc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6827","nonce":193784582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6829","nonce":193784582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6833","nonce":193784582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6831","nonce":193784582}]},"public_addr":"172.21.15.89:6827/193784582","cluster_addr":"172.21.15.89:6829/193784582","heartbeat_back_addr":"172.21.15.89:6833/193784582","heartbeat_front_addr":"172.21.15.89:6831/193784582","state":["exists","up"]},{"osd":4,"uuid":"3d743da7-8b8c-40f8-a591-e791ed280233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6801","nonce":2252846533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6803","nonce":2252846533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6807","nonce":2252846533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6805","nonce":2252846533}]},"public_addr":"172.21.15.152:6801/2252846533","cluster_addr":"172.21.15.152:6803/2252846533","heartbeat_back_addr":"172.21.15.152:6807/2252846533","heartbeat_front_addr":"172.21.15.152:6805/2252846533","state":["exists","up"]},{"osd":5,"uuid":"e2ece7d8-4a4a-4247-b3af-46c854bcc1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6809","nonce":3145783803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6811","nonce":3145783803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6815","nonce":3145783803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6813","nonce":3145783803}]},"public_addr":"172.21.15.152:6809/3145783803","cluster_addr":"172.21.15.152:6811/3145783803","heartbeat_back_addr":"172.21.15.152:6815/3145783803","heartbeat_front_addr":"172.21.15.152:6813/3145783803","state":["exists","up"]},{"osd":6,"uuid":"877ec6ea-e165-4d53-aa88-87004c5b26c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6817","nonce":3295577655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6819","nonce":3295577655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6823","nonce":3295577655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6821","nonce":3295577655}]},"public_addr":"172.21.15.152:6817/3295577655","cluster_addr":"172.21.15.152:6819/3295577655","heartbeat_back_addr":"172.21.15.152:6823/3295577655","heartbeat_front_addr":"172.21.15.152:6821/3295577655","state":["exists","up"]},{"osd":7,"uuid":"787eb11f-c37b-4862-b1e7-485a9da2851e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6825","nonce":440724144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6827","nonce":440724144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6831","nonce":440724144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6829","nonce":440724144}]},"public_addr":"172.21.15.152:6825/440724144","cluster_addr":"172.21.15.152:6827/440724144","heartbeat_back_addr":"172.21.15.152:6831/440724144","heartbeat_front_addr":"172.21.15.152:6829/440724144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:07.805982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:32.484356+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:57.086765+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:09:21.879472+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:09:44.830909+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:10:10.422590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:10:35.568456+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:11:00.893882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/745558169":"2024-03-22T13:06:42.905029+0000","172.21.15.89:6801/1022748022":"2024-03-22T13:06:42.905029+0000","172.21.15.89:6800/1022748022":"2024-03-22T13:06:42.905029+0000","172.21.15.89:0/3750854419":"2024-03-22T13:06:16.698081+0000","172.21.15.89:6800/3099145381":"2024-03-22T13:06:16.698081+0000","172.21.15.89:6801/3099145381":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/1302626015":"2024-03-22T13:06:42.905029+0000","172.21.15.89:0/346547163":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/249140509":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/586007047":"2024-03-22T13:06:42.905029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T13:11:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 msgr2=0x7ff1b80707f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T13:11:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 0x7ff1b80707f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 msgr2=0x7ff1cc074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 0x7ff1cc074490 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800eb30 tx=0x7ff1c800ee40 comp rx=0 tx=0).stop 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 shutdown_connections 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:6800/4277445380,v1:172.21.15.89:6801/4277445380] conn(0x7ff1b806e330 0x7ff1b80707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1cc1074b0 0x7ff1cc074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc1023d0 0x7ff1cc074490 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 --2- 172.21.15.89:0/4289576095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc102d80 0x7ff1cc0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 >> 172.21.15.89:0/4289576095 conn(0x7ff1cc0fdc40 msgr2=0x7ff1cc0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 shutdown_connections 2024-03-21T13:11:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:54.536+0000 7ff1d40d9700 1 -- 172.21.15.89:0/4289576095 wait complete. 2024-03-21T13:11:54.725 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:54.391+0000 7f74a6917200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:11:54.726 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:54.493+0000 7f74a6917200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:11:54.726 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:54.725+0000 7f74a6917200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:11:54.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[39209]: Standby manager daemon x restarted 2024-03-21T13:11:54.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[39209]: Standby manager daemon x started 2024-03-21T13:11:54.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:11:54.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:11:54.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4289576095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[39209]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[30095]: Standby manager daemon x restarted 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[30095]: Standby manager daemon x started 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4289576095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:54.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-mon[30095]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-mon[34051]: Standby manager daemon x restarted 2024-03-21T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-mon[34051]: Standby manager daemon x started 2024-03-21T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4289576095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:54.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-mon[34051]: from='mgr.? 172.21.15.152:0/2273861974' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:11:54.822 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: [21/Mar/2024:13:11:54] ENGINE Bus STARTING 2024-03-21T13:11:54.822 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: CherryPy Checker: 2024-03-21T13:11:54.822 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: The Application mounted at '' has an empty config. 2024-03-21T13:11:54.822 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: 2024-03-21T13:11:54.822 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: [21/Mar/2024:13:11:54] ENGINE Serving on http://:::9283 2024-03-21T13:11:54.822 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:11:54 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x[36131]: [21/Mar/2024:13:11:54] ENGINE Bus STARTED 2024-03-21T13:11:54.983 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:11:54] ENGINE Bus STARTING 2024-03-21T13:11:54.983 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: CherryPy Checker: 2024-03-21T13:11:54.983 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: The Application mounted at '' has an empty config. 2024-03-21T13:11:54.984 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:11:54.984 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:11:54] ENGINE Serving on http://:::9283 2024-03-21T13:11:54.984 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:54 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:11:54] ENGINE Bus STARTED 2024-03-21T13:11:55.426 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T13:11:55.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T13:11:55.694 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:55.818 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T13:11:55.818 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: Active manager daemon y restarted 2024-03-21T13:11:55.818 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: Activating manager daemon y 2024-03-21T13:11:55.818 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: mgrmap e19: y(active, starting, since 0.00863356s), standbys: x 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:11:55.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: Manager daemon y is now available 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:11:55.820 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: Active manager daemon y restarted 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: Activating manager daemon y 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: mgrmap e19: y(active, starting, since 0.00863356s), standbys: x 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:11:55.821 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: Manager daemon y is now available 2024-03-21T13:11:55.822 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:55.823 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:55.823 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:55.823 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:11:55.823 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:11:55.823 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:11:55.823 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:55 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:11:55.988 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T13:11:55.988 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: Active manager daemon y restarted 2024-03-21T13:11:55.988 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: Activating manager daemon y 2024-03-21T13:11:55.988 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: mgrmap e19: y(active, starting, since 0.00863356s), standbys: x 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:11:55.989 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: Manager daemon y is now available 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:11:55.990 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:55 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:11:56.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:56 smithi152 ceph-mon[34051]: [21/Mar/2024:13:11:55] ENGINE Bus STARTING 2024-03-21T13:11:56.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:56 smithi152 ceph-mon[34051]: [21/Mar/2024:13:11:55] ENGINE Serving on https://172.21.15.89:7150 2024-03-21T13:11:56.959 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:56 smithi152 ceph-mon[34051]: [21/Mar/2024:13:11:55] ENGINE Serving on http://172.21.15.89:8765 2024-03-21T13:11:56.959 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:56 smithi152 ceph-mon[34051]: [21/Mar/2024:13:11:55] ENGINE Bus STARTED 2024-03-21T13:11:56.959 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:56 smithi152 ceph-mon[34051]: mgrmap e20: y(active, since 1.01198s), standbys: x 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[30095]: [21/Mar/2024:13:11:55] ENGINE Bus STARTING 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[30095]: [21/Mar/2024:13:11:55] ENGINE Serving on https://172.21.15.89:7150 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[30095]: [21/Mar/2024:13:11:55] ENGINE Serving on http://172.21.15.89:8765 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[30095]: [21/Mar/2024:13:11:55] ENGINE Bus STARTED 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[30095]: mgrmap e20: y(active, since 1.01198s), standbys: x 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[39209]: [21/Mar/2024:13:11:55] ENGINE Bus STARTING 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[39209]: [21/Mar/2024:13:11:55] ENGINE Serving on https://172.21.15.89:7150 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[39209]: [21/Mar/2024:13:11:55] ENGINE Serving on http://172.21.15.89:8765 2024-03-21T13:11:57.014 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[39209]: [21/Mar/2024:13:11:55] ENGINE Bus STARTED 2024-03-21T13:11:57.015 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:56 smithi089 ceph-mon[39209]: mgrmap e20: y(active, since 1.01198s), standbys: x 2024-03-21T13:11:57.575 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:11:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:11:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:11:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/2325174307 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f8102ba0 msgr2=0x7fd6f810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/2325174307 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f8102ba0 0x7fd6f810efe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000b0a0 tx=0x7fd6f000b3b0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/2325174307 shutdown_connections 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/2325174307 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f8102ba0 0x7fd6f810efe0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/2325174307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6f81081c0 0x7fd6f81085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/2325174307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6f81021e0 0x7fd6f8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/2325174307 >> 172.21.15.89:0/2325174307 conn(0x7fd6f80792c0 msgr2=0x7fd6f80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/2325174307 shutdown_connections 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.585+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/2325174307 wait complete. 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 Processor -- start 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 -- start start 2024-03-21T13:11:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f81021e0 0x7fd6f8197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6f8102ba0 0x7fd6f81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6f81081c0 0x7fd6f819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f81118f0 con 0x7fd6f81081c0 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8111770 con 0x7fd6f8102ba0 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6fd1aa700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8111a70 con 0x7fd6f81021e0 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f81021e0 0x7fd6f8197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f759e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6f81081c0 0x7fd6f819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f81021e0 0x7fd6f8197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:46936/0 (socket says 172.21.15.89:46936) 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 -- 172.21.15.89:0/1909408154 learned_addr learned my addr 172.21.15.89:0/1909408154 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f659c700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6f8102ba0 0x7fd6f81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 -- 172.21.15.89:0/1909408154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6f8102ba0 msgr2=0x7fd6f81982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6f8102ba0 0x7fd6f81982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 -- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6f81081c0 msgr2=0x7fd6f819c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6f81081c0 0x7fd6f819c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f6d9d700 1 -- 172.21.15.89:0/1909408154 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f819cd80 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.586+0000 7fd6f759e700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6f81081c0 0x7fd6f819c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.587+0000 7fd6f6d9d700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f81021e0 0x7fd6f8197d70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800d980 tx=0x7fd6e800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.587+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8017070 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.587+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f819d070 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.587+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd6e8005020 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.587+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e800f990 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.587+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f81a48f0 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.588+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f8068e50 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.588+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 20) v1 ==== 91939+0+0 (secure 0 0 0) 0x7fd6e800fb30 con 0x7fd6f81021e0 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.589+0000 7fd6dffff700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fd6e006e050 0x7fd6e0070510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:11:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.589+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fd6e808e1f0 con 0x7fd6f81021e0 2024-03-21T13:11:57.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.589+0000 7fd6f659c700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fd6e006e050 0x7fd6e0070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:11:57.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.589+0000 7fd6f659c700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fd6e006e050 0x7fd6e0070510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009680 tx=0x7fd6ec009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:11:57.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.596+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd6e8020020 con 0x7fd6f81021e0 2024-03-21T13:11:57.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.756+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mgrmap(e 21) v1 ==== 92131+0+0 (secure 0 0 0) 0x7fd6e80054b0 con 0x7fd6f81021e0 2024-03-21T13:11:57.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.800+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6f81a4ac0 con 0x7fd6f81021e0 2024-03-21T13:11:57.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.802+0000 7fd6dffff700 1 -- 172.21.15.89:0/1909408154 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+23161 (secure 0 0 0) 0x7fd6e8022300 con 0x7fd6f81021e0 2024-03-21T13:11:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:11:57.803 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"fsid":"9c355834-e783-11ee-95cd-87774f69a715","created":"2024-03-21T13:05:40.974140+0000","modified":"2024-03-21T13:11:54.739515+0000","last_up_change":"2024-03-21T13:11:02.379434+0000","last_in_change":"2024-03-21T13:10:44.727962+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T13:09:01.242029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T13:11:14.467225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T13:11:15.118984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T13:11:16.456191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T13:11:18.540730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T13:11:20.611141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4258f99e-44cf-4cd0-a093-00e1192a8ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6803","nonce":1699462714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6805","nonce":1699462714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6809","nonce":1699462714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1699462714},{"type":"v1","addr":"172.21.15.89:6807","nonce":1699462714}]},"public_addr":"172.21.15.89:6803/1699462714","cluster_addr":"172.21.15.89:6805/1699462714","heartbeat_back_addr":"172.21.15.89:6809/1699462714","heartbeat_front_addr":"172.21.15.89:6807/1699462714","state":["exists","up"]},{"osd":1,"uuid":"debf8a73-c369-49ed-8886-57f2ca61e6d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6811","nonce":3822115704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6813","nonce":3822115704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6817","nonce":3822115704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":3822115704},{"type":"v1","addr":"172.21.15.89:6815","nonce":3822115704}]},"public_addr":"172.21.15.89:6811/3822115704","cluster_addr":"172.21.15.89:6813/3822115704","heartbeat_back_addr":"172.21.15.89:6817/3822115704","heartbeat_front_addr":"172.21.15.89:6815/3822115704","state":["exists","up"]},{"osd":2,"uuid":"2686770f-bbbc-4082-bb00-9f6f10f8783b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6819","nonce":2934798428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6821","nonce":2934798428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6825","nonce":2934798428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2934798428},{"type":"v1","addr":"172.21.15.89:6823","nonce":2934798428}]},"public_addr":"172.21.15.89:6819/2934798428","cluster_addr":"172.21.15.89:6821/2934798428","heartbeat_back_addr":"172.21.15.89:6825/2934798428","heartbeat_front_addr":"172.21.15.89:6823/2934798428","state":["exists","up"]},{"osd":3,"uuid":"4c996867-ce38-4ae7-b934-5faf2ce2fc0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6827","nonce":193784582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6829","nonce":193784582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6833","nonce":193784582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":193784582},{"type":"v1","addr":"172.21.15.89:6831","nonce":193784582}]},"public_addr":"172.21.15.89:6827/193784582","cluster_addr":"172.21.15.89:6829/193784582","heartbeat_back_addr":"172.21.15.89:6833/193784582","heartbeat_front_addr":"172.21.15.89:6831/193784582","state":["exists","up"]},{"osd":4,"uuid":"3d743da7-8b8c-40f8-a591-e791ed280233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6801","nonce":2252846533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6803","nonce":2252846533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6807","nonce":2252846533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2252846533},{"type":"v1","addr":"172.21.15.152:6805","nonce":2252846533}]},"public_addr":"172.21.15.152:6801/2252846533","cluster_addr":"172.21.15.152:6803/2252846533","heartbeat_back_addr":"172.21.15.152:6807/2252846533","heartbeat_front_addr":"172.21.15.152:6805/2252846533","state":["exists","up"]},{"osd":5,"uuid":"e2ece7d8-4a4a-4247-b3af-46c854bcc1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6809","nonce":3145783803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6811","nonce":3145783803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6815","nonce":3145783803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3145783803},{"type":"v1","addr":"172.21.15.152:6813","nonce":3145783803}]},"public_addr":"172.21.15.152:6809/3145783803","cluster_addr":"172.21.15.152:6811/3145783803","heartbeat_back_addr":"172.21.15.152:6815/3145783803","heartbeat_front_addr":"172.21.15.152:6813/3145783803","state":["exists","up"]},{"osd":6,"uuid":"877ec6ea-e165-4d53-aa88-87004c5b26c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6817","nonce":3295577655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6819","nonce":3295577655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6823","nonce":3295577655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3295577655},{"type":"v1","addr":"172.21.15.152:6821","nonce":3295577655}]},"public_addr":"172.21.15.152:6817/3295577655","cluster_addr":"172.21.15.152:6819/3295577655","heartbeat_back_addr":"172.21.15.152:6823/3295577655","heartbeat_front_addr":"172.21.15.152:6821/3295577655","state":["exists","up"]},{"osd":7,"uuid":"787eb11f-c37b-4862-b1e7-485a9da2851e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6825","nonce":440724144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6827","nonce":440724144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6831","nonce":440724144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":440724144},{"type":"v1","addr":"172.21.15.152:6829","nonce":440724144}]},"public_addr":"172.21.15.152:6825/440724144","cluster_addr":"172.21.15.152:6827/440724144","heartbeat_back_addr":"172.21.15.152:6831/440724144","heartbeat_front_addr":"172.21.15.152:6829/440724144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:07.805982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:32.484356+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:08:57.086765+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:09:21.879472+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:09:44.830909+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:10:10.422590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:10:35.568456+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T13:11:00.893882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/197165808":"2024-03-22T13:11:54.739469+0000","172.21.15.89:0/319612939":"2024-03-22T13:11:54.739469+0000","172.21.15.89:0/346547163":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/586007047":"2024-03-22T13:06:42.905029+0000","172.21.15.89:6801/3099145381":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/249140509":"2024-03-22T13:06:16.698081+0000","172.21.15.89:6800/3099145381":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/3750854419":"2024-03-22T13:06:16.698081+0000","172.21.15.89:0/1302626015":"2024-03-22T13:06:42.905029+0000","172.21.15.89:0/26756681":"2024-03-22T13:11:54.739469+0000","172.21.15.89:6800/1022748022":"2024-03-22T13:06:42.905029+0000","172.21.15.89:0/745558169":"2024-03-22T13:06:42.905029+0000","172.21.15.89:6800/4277445380":"2024-03-22T13:11:54.739469+0000","172.21.15.89:6801/1022748022":"2024-03-22T13:06:42.905029+0000","172.21.15.89:0/4124817476":"2024-03-22T13:11:54.739469+0000","172.21.15.89:6801/4277445380":"2024-03-22T13:11:54.739469+0000","172.21.15.89:0/391889855":"2024-03-22T13:11:54.739469+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fd6e006e050 msgr2=0x7fd6e0070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fd6e006e050 0x7fd6e0070510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009680 tx=0x7fd6ec009260 comp rx=0 tx=0).stop 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f81021e0 msgr2=0x7fd6f8197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f81021e0 0x7fd6f8197d70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800d980 tx=0x7fd6e800dc90 comp rx=0 tx=0).stop 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 shutdown_connections 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6f81081c0 0x7fd6f819c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fd6e006e050 0x7fd6e0070510 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6f81021e0 0x7fd6f8197d70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 --2- 172.21.15.89:0/1909408154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6f8102ba0 0x7fd6f81982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:11:57.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 >> 172.21.15.89:0/1909408154 conn(0x7fd6f80792c0 msgr2=0x7fd6f8105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:11:57.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 shutdown_connections 2024-03-21T13:11:57.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:11:57.804+0000 7fd6fd1aa700 1 -- 172.21.15.89:0/1909408154 wait complete. 2024-03-21T13:11:57.871 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:57 smithi152 ceph-mon[34051]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:57.871 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:11:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:11:57.945 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:57 smithi089 ceph-mon[30095]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:57.945 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:57 smithi089 ceph-mon[39209]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:58.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T13:11:58.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T13:11:58.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T13:11:58.603 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T13:11:58.603 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T13:11:58.604 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T13:11:58.604 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T13:11:58.604 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T13:11:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[30095]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[30095]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-21T13:11:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1909408154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[39209]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[39209]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-21T13:11:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1909408154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:11:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:58 smithi152 ceph-mon[34051]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:11:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:58 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:11:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:58 smithi152 ceph-mon[34051]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-21T13:11:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1909408154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T13:11:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:11:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:11:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:11:59.192 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:12:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi152 to 3760M 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[30095]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.312 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi152 to 3760M 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.313 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:00 smithi089 ceph-mon[39209]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T13:12:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:12:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:00.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi152 to 3760M 2024-03-21T13:12:00.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:00.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:00 smithi152 ceph-mon[34051]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T13:12:02.424 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:02 smithi089 ceph-mon[30095]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:12:02.424 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:02 smithi089 ceph-mon[39209]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:12:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:02 smithi152 ceph-mon[34051]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:12:03.425 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:03 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:03.425 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:03 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:03.796 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:03 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:03.797 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:03 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:03.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:03 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:03.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:03 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.038 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:04.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T13:12:04.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.725 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:12:04.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:04 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:12:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T13:12:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:12:04.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:04 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi089 to 3760M 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.038 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi089 to 3760M 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:12:06.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:12:06.040 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:12:06.040 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.040 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.040 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.040 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.040 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:05 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi089 to 3760M 2024-03-21T13:12:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-21T13:12:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-21T13:12:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: Updating smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:05 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 -- 172.21.15.89:0/4164743307 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e81070d0 msgr2=0x7f45e81074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 --2- 172.21.15.89:0/4164743307 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e81070d0 0x7f45e81074b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009990 tx=0x7f45d8009ca0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 -- 172.21.15.89:0/4164743307 shutdown_connections 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 --2- 172.21.15.89:0/4164743307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45e8101a30 0x7f45e810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 --2- 172.21.15.89:0/4164743307 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e81070d0 0x7f45e81074b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 --2- 172.21.15.89:0/4164743307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45e8101090 0x7f45e81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 -- 172.21.15.89:0/4164743307 >> 172.21.15.89:0/4164743307 conn(0x7f45e8077fc0 msgr2=0x7f45e80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 -- 172.21.15.89:0/4164743307 shutdown_connections 2024-03-21T13:12:06.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.423+0000 7f45ec900700 1 -- 172.21.15.89:0/4164743307 wait complete. 2024-03-21T13:12:06.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 Processor -- start 2024-03-21T13:12:06.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 -- start start 2024-03-21T13:12:06.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e8101090 0x7f45e8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:06.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45e8101a30 0x7f45e81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45e81070d0 0x7f45e819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8110840 con 0x7f45e81070d0 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81106c0 con 0x7f45e8101a30 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ec900700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81109c0 con 0x7f45e8101090 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45e659c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e8101090 0x7f45e8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45e659c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e8101090 0x7f45e8197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:35902/0 (socket says 172.21.15.89:35902) 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45e659c700 1 -- 172.21.15.89:0/72337209 learned_addr learned my addr 172.21.15.89:0/72337209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45e6d9d700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45e81070d0 0x7f45e819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.424+0000 7f45ddd9b700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45e8101a30 0x7f45e81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45e659c700 1 -- 172.21.15.89:0/72337209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45e8101a30 msgr2=0x7f45e81983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45e659c700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45e8101a30 0x7f45e81983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45e659c700 1 -- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45e81070d0 msgr2=0x7f45e819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:06.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45e659c700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45e81070d0 0x7f45e819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45e659c700 1 -- 172.21.15.89:0/72337209 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e819ce70 con 0x7f45e8101090 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45e6d9d700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45e81070d0 0x7f45e819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45e659c700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e8101090 0x7f45e8197eb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f45d8000c00 tx=0x7f45d80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8027070 con 0x7f45e8101090 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f45d8022470 con 0x7f45e8101090 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8017650 con 0x7f45e8101090 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e819d100 con 0x7f45e8101090 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.425+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e81a49e0 con 0x7f45e8101090 2024-03-21T13:12:06.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.426+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45e8100700 con 0x7f45e8101090 2024-03-21T13:12:06.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.426+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f45d802a070 con 0x7f45e8101090 2024-03-21T13:12:06.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45dffff700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f45d406e340 0x7f45d4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:06.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f45d808f870 con 0x7f45e8101090 2024-03-21T13:12:06.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45ddd9b700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f45d406e340 0x7f45d4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:06.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45dffff700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] conn(0x7f45d4078250 0x7f45d407a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:06.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 --> [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45d80043d0 con 0x7f45d4078250 2024-03-21T13:12:06.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45d80942c0 con 0x7f45e8101090 2024-03-21T13:12:06.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45e6d9d700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] conn(0x7f45d4078250 0x7f45d407a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:06.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45ddd9b700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f45d406e340 0x7f45d4070800 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45e80786a0 tx=0x7f45d0009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:06.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.427+0000 7f45e6d9d700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] conn(0x7f45d4078250 0x7f45d407a6f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:06.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.428+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== osd.2 v2:172.21.15.89:6818/2934798428 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f45d80043d0 con 0x7f45d4078250 2024-03-21T13:12:06.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.444+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 --> [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f45e8102780 con 0x7f45d4078250 2024-03-21T13:12:06.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45dffff700 1 -- 172.21.15.89:0/72337209 <== osd.2 v2:172.21.15.89:6818/2934798428 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f45e8102780 con 0x7f45d4078250 2024-03-21T13:12:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] conn(0x7f45d4078250 msgr2=0x7f45d407a6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6818/2934798428,v1:172.21.15.89:6819/2934798428] conn(0x7f45d4078250 0x7f45d407a6f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f45d406e340 msgr2=0x7f45d4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f45d406e340 0x7f45d4070800 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45e80786a0 tx=0x7f45d0009260 comp rx=0 tx=0).stop 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e8101090 msgr2=0x7f45e8197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 --2- 172.21.15.89:0/72337209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f45e8101090 0x7f45e8197eb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f45d8000c00 tx=0x7f45d80047b0 comp rx=0 tx=0).stop 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45e6d9d700 1 -- 172.21.15.89:0/72337209 reap_dead start 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 shutdown_connections 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 >> 172.21.15.89:0/72337209 conn(0x7f45e8077fc0 msgr2=0x7f45e80ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 shutdown_connections 2024-03-21T13:12:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:06.445+0000 7f45ec900700 1 -- 172.21.15.89:0/72337209 wait complete. 2024-03-21T13:12:06.984 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:06 smithi089 ceph-mon[39209]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T13:12:06.984 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:06 smithi089 ceph-mon[39209]: Deploying daemon alertmanager.a on smithi089 2024-03-21T13:12:06.984 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:06 smithi089 ceph-mon[30095]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T13:12:06.984 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:06 smithi089 ceph-mon[30095]: Deploying daemon alertmanager.a on smithi089 2024-03-21T13:12:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:06 smithi152 ceph-mon[34051]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T13:12:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:06 smithi152 ceph-mon[34051]: Deploying daemon alertmanager.a on smithi089 2024-03-21T13:12:07.757 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:12:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:07.758 INFO:teuthology.orchestra.run.smithi089.stdout:77309411368 2024-03-21T13:12:07.758 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T13:12:08.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:12:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:12:08.991 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:08 smithi089 ceph-mon[30095]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T13:12:08.991 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:08 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:08.992 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:08 smithi089 ceph-mon[39209]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T13:12:08.992 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:08 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:08 smithi152 ceph-mon[34051]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T13:12:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:08 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:09.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 -- 172.21.15.89:0/3028004894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 msgr2=0x7f3aa8108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 --2- 172.21.15.89:0/3028004894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 0x7f3aa8108690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009990 tx=0x7f3a98009ca0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 -- 172.21.15.89:0/3028004894 shutdown_connections 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 --2- 172.21.15.89:0/3028004894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa8102c00 0x7f3aa810eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 --2- 172.21.15.89:0/3028004894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 0x7f3aa8108690 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 --2- 172.21.15.89:0/3028004894 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8102240 0x7f3aa81026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 -- 172.21.15.89:0/3028004894 >> 172.21.15.89:0/3028004894 conn(0x7f3aa8079260 msgr2=0x7f3aa8079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 -- 172.21.15.89:0/3028004894 shutdown_connections 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.620+0000 7f3aafc0c700 1 -- 172.21.15.89:0/3028004894 wait complete. 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 Processor -- start 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 -- start start 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa8102240 0x7f3aa8197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8102c00 0x7f3aa8198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 0x7f3aa819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8111950 con 0x7f3aa81082b0 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa81117d0 con 0x7f3aa8102240 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aafc0c700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8111ad0 con 0x7f3aa8102c00 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aae1a9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 0x7f3aa819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aad1a7700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8102c00 0x7f3aa8198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aae1a9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 0x7f3aa819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53508/0 (socket says 172.21.15.89:53508) 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aae1a9700 1 -- 172.21.15.89:0/2110878111 learned_addr learned my addr 172.21.15.89:0/2110878111 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.621+0000 7f3aad9a8700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa8102240 0x7f3aa8197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aad1a7700 1 -- 172.21.15.89:0/2110878111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa8102240 msgr2=0x7f3aa8197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aad1a7700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa8102240 0x7f3aa8197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aad1a7700 1 -- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 msgr2=0x7f3aa819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aad1a7700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 0x7f3aa819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aad1a7700 1 -- 172.21.15.89:0/2110878111 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa819cdd0 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aae1a9700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa81082b0 0x7f3aa819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aad9a8700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa8102240 0x7f3aa8197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aad1a7700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8102c00 0x7f3aa8198350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400e9f0 tx=0x7f3aa400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa400cbd0 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3aa400cd70 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4018680 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa819d0c0 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.622+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa80743c0 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.623+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3aa8107780 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.623+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f3aa4010430 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3a9effd700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f3a9406e340 0x7f3a94070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f3aa408ea10 con 0x7f3aa8102c00 2024-03-21T13:12:09.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3aad9a8700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f3a9406e340 0x7f3a94070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3a9effd700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] conn(0x7f3a940781d0 0x7f3a9407a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 --> [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3aa4012ec0 con 0x7f3a940781d0 2024-03-21T13:12:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3aa40932c0 con 0x7f3aa8102c00 2024-03-21T13:12:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3aae1a9700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] conn(0x7f3a940781d0 0x7f3a9407a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3aad9a8700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f3a9406e340 0x7f3a94070800 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006010 tx=0x7f3a9801a040 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.624+0000 7f3aae1a9700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] conn(0x7f3a940781d0 0x7f3a9407a670 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.625+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== osd.1 v2:172.21.15.89:6810/3822115704 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3aa4012ec0 con 0x7f3a940781d0 2024-03-21T13:12:09.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 --> [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3aa81038d0 con 0x7f3a940781d0 2024-03-21T13:12:09.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3a9effd700 1 -- 172.21.15.89:0/2110878111 <== osd.1 v2:172.21.15.89:6810/3822115704 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3aa81038d0 con 0x7f3a940781d0 2024-03-21T13:12:09.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] conn(0x7f3a940781d0 msgr2=0x7f3a9407a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6810/3822115704,v1:172.21.15.89:6811/3822115704] conn(0x7f3a940781d0 0x7f3a9407a670 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f3a9406e340 msgr2=0x7f3a94070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f3a9406e340 0x7f3a94070800 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006010 tx=0x7f3a9801a040 comp rx=0 tx=0).stop 2024-03-21T13:12:09.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8102c00 msgr2=0x7f3aa8198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 --2- 172.21.15.89:0/2110878111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8102c00 0x7f3aa8198350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400e9f0 tx=0x7f3aa400ed00 comp rx=0 tx=0).stop 2024-03-21T13:12:09.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aae1a9700 1 -- 172.21.15.89:0/2110878111 reap_dead start 2024-03-21T13:12:09.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 shutdown_connections 2024-03-21T13:12:09.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.642+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 >> 172.21.15.89:0/2110878111 conn(0x7f3aa8079260 msgr2=0x7f3aa8105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:09.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.643+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 shutdown_connections 2024-03-21T13:12:09.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.643+0000 7f3aafc0c700 1 -- 172.21.15.89:0/2110878111 wait complete. 2024-03-21T13:12:09.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 -- 172.21.15.89:0/1164419055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 msgr2=0x7f633c10d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 --2- 172.21.15.89:0/1164419055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 0x7f633c10d720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f633400b0a0 tx=0x7f633400b3b0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 -- 172.21.15.89:0/1164419055 shutdown_connections 2024-03-21T13:12:09.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 --2- 172.21.15.89:0/1164419055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 0x7f633c10d720 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 --2- 172.21.15.89:0/1164419055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f633c106850 0x7f633c106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 --2- 172.21.15.89:0/1164419055 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c100850 0x7f633c100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 -- 172.21.15.89:0/1164419055 >> 172.21.15.89:0/1164419055 conn(0x7f633c077690 msgr2=0x7f633c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 -- 172.21.15.89:0/1164419055 shutdown_connections 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.512+0000 7f63440ec700 1 -- 172.21.15.89:0/1164419055 wait complete. 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 Processor -- start 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 -- start start 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f633c100850 0x7f633c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 0x7f633c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c106850 0x7f633c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c10f7c0 con 0x7f633c101210 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c10f640 con 0x7f633c100850 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f63440ec700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c10f940 con 0x7f633c106850 2024-03-21T13:12:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f6342689700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c106850 0x7f633c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f6342689700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c106850 0x7f633c19e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:51396/0 (socket says 172.21.15.89:51396) 2024-03-21T13:12:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f6342689700 1 -- 172.21.15.89:0/1529224124 learned_addr learned my addr 172.21.15.89:0/1529224124 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f6341687700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 0x7f633c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.513+0000 7f6341e88700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f633c100850 0x7f633c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6342689700 1 -- 172.21.15.89:0/1529224124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f633c100850 msgr2=0x7f633c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6342689700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f633c100850 0x7f633c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6342689700 1 -- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 msgr2=0x7f633c19a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6342689700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 0x7f633c19a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6342689700 1 -- 172.21.15.89:0/1529224124 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c19efe0 con 0x7f633c106850 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6341687700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633c101210 0x7f633c19a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6342689700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c106850 0x7f633c19e8a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f633400b070 tx=0x7f6334007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334021020 con 0x7f633c106850 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c19f270 con 0x7f633c106850 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6334004c50 con 0x7f633c106850 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334005050 con 0x7f633c106850 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.514+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c1a6b50 con 0x7f633c106850 2024-03-21T13:12:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.515+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f633c104120 con 0x7f633c106850 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.516+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f6334024020 con 0x7f633c106850 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.516+0000 7f6332ffd700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f632806e340 0x7f6328070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.516+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f633400f020 con 0x7f633c106850 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.516+0000 7f6341e88700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f632806e340 0x7f6328070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.516+0000 7f6332ffd700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] conn(0x7f63280781d0 0x7f632807a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.516+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 --> [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f632807ad40 con 0x7f63280781d0 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.516+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6334090150 con 0x7f633c106850 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.517+0000 7f6341687700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] conn(0x7f63280781d0 0x7f632807a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.517+0000 7f6341e88700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f632806e340 0x7f6328070800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f633800abd0 tx=0x7f633800a380 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.517+0000 7f6341687700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] conn(0x7f63280781d0 0x7f632807a670 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:09.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.518+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== osd.3 v2:172.21.15.89:6826/193784582 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f632807ad40 con 0x7f63280781d0 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.534+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 --> [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f633c101f90 con 0x7f63280781d0 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.534+0000 7f6332ffd700 1 -- 172.21.15.89:0/1529224124 <== osd.3 v2:172.21.15.89:6826/193784582 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f633c101f90 con 0x7f63280781d0 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.534+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] conn(0x7f63280781d0 msgr2=0x7f632807a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.534+0000 7f63440ec700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6826/193784582,v1:172.21.15.89:6827/193784582] conn(0x7f63280781d0 0x7f632807a670 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f632806e340 msgr2=0x7f6328070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f632806e340 0x7f6328070800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f633800abd0 tx=0x7f633800a380 comp rx=0 tx=0).stop 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c106850 msgr2=0x7f633c19e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 --2- 172.21.15.89:0/1529224124 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c106850 0x7f633c19e8a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f633400b070 tx=0x7f6334007a20 comp rx=0 tx=0).stop 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f6342689700 1 -- 172.21.15.89:0/1529224124 reap_dead start 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 shutdown_connections 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 >> 172.21.15.89:0/1529224124 conn(0x7f633c077690 msgr2=0x7f633c103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 shutdown_connections 2024-03-21T13:12:09.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:09.535+0000 7f63440ec700 1 -- 172.21.15.89:0/1529224124 wait complete. 2024-03-21T13:12:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 -- 172.21.15.89:0/1951214888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101090 msgr2=0x7f7fac1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 --2- 172.21.15.89:0/1951214888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101090 0x7f7fac1014f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009990 tx=0x7f7fa8009ca0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 -- 172.21.15.89:0/1951214888 shutdown_connections 2024-03-21T13:12:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 --2- 172.21.15.89:0/1951214888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fac101a30 0x7f7fac10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 --2- 172.21.15.89:0/1951214888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fac1070d0 0x7f7fac1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 --2- 172.21.15.89:0/1951214888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101090 0x7f7fac1014f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 -- 172.21.15.89:0/1951214888 >> 172.21.15.89:0/1951214888 conn(0x7f7fac077fc0 msgr2=0x7f7fac0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 -- 172.21.15.89:0/1951214888 shutdown_connections 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.086+0000 7f7fb4351700 1 -- 172.21.15.89:0/1951214888 wait complete. 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 Processor -- start 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 -- start start 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fac101090 0x7f7fac19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101a30 0x7f7fac19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fac1070d0 0x7f7fac196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac110800 con 0x7f7fac101a30 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac110680 con 0x7f7fac101090 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb4351700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac110980 con 0x7f7fac1070d0 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb18ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101a30 0x7f7fac19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb28ee700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fac1070d0 0x7f7fac196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb28ee700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fac1070d0 0x7f7fac196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:35934/0 (socket says 172.21.15.89:35934) 2024-03-21T13:12:10.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb28ee700 1 -- 172.21.15.89:0/3833056114 learned_addr learned my addr 172.21.15.89:0/3833056114 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:10.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.087+0000 7f7fb20ed700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fac101090 0x7f7fac19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:10.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb18ec700 1 -- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fac1070d0 msgr2=0x7f7fac196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:10.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb18ec700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fac1070d0 0x7f7fac196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb18ec700 1 -- 172.21.15.89:0/3833056114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fac101090 msgr2=0x7f7fac19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb18ec700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fac101090 0x7f7fac19c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb18ec700 1 -- 172.21.15.89:0/3833056114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac197250 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb28ee700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fac1070d0 0x7f7fac196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb18ec700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101a30 0x7f7fac19ce70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80059d0 tx=0x7f7fa80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa802a070 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac1974e0 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7fa800f460 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8025680 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.088+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac069e40 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.089+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7fac106670 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.089+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f7fa802d070 con 0x7f7fac101a30 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fa37fe700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7f9806e340 0x7f7f98070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f7fa8092230 con 0x7f7fac101a30 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fb20ed700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7f9806e340 0x7f7f98070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fa37fe700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] conn(0x7f7f980781d0 0x7f7f9807a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 --> [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fa80056d0 con 0x7f7f980781d0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fa80926c0 con 0x7f7fac101a30 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fb28ee700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] conn(0x7f7f980781d0 0x7f7f9807a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fb20ed700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7f9806e340 0x7f7f98070800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009680 tx=0x7f7f9c009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.090+0000 7f7fb28ee700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] conn(0x7f7f980781d0 0x7f7f9807a670 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.091+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== osd.0 v2:172.21.15.89:6802/1699462714 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7fa80056d0 con 0x7f7f980781d0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.107+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 --> [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7fac102780 con 0x7f7f980781d0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.107+0000 7f7fa37fe700 1 -- 172.21.15.89:0/3833056114 <== osd.0 v2:172.21.15.89:6802/1699462714 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7fac102780 con 0x7f7f980781d0 2024-03-21T13:12:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] conn(0x7f7f980781d0 msgr2=0x7f7f9807a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6802/1699462714,v1:172.21.15.89:6803/1699462714] conn(0x7f7f980781d0 0x7f7f9807a670 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7f9806e340 msgr2=0x7f7f98070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7f9806e340 0x7f7f98070800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009680 tx=0x7f7f9c009260 comp rx=0 tx=0).stop 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101a30 msgr2=0x7f7fac19ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 --2- 172.21.15.89:0/3833056114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fac101a30 0x7f7fac19ce70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80059d0 tx=0x7f7fa80047b0 comp rx=0 tx=0).stop 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb28ee700 1 -- 172.21.15.89:0/3833056114 reap_dead start 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 shutdown_connections 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 >> 172.21.15.89:0/3833056114 conn(0x7f7fac077fc0 msgr2=0x7f7fac104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 shutdown_connections 2024-03-21T13:12:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:07.108+0000 7f7fb4351700 1 -- 172.21.15.89:0/3833056114 wait complete. 2024-03-21T13:12:10.860 INFO:teuthology.orchestra.run.smithi089.stdout:34359738418 2024-03-21T13:12:10.861 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T13:12:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:10 smithi152 ceph-mon[34051]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:10 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:11.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:10 smithi089 ceph-mon[30095]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:11.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:10 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:11.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:10 smithi089 ceph-mon[39209]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:11.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:10 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:11.343 INFO:teuthology.orchestra.run.smithi089.stdout:55834574893 2024-03-21T13:12:11.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T13:12:11.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 -- 172.21.15.89:0/3446666430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a14103dc0 msgr2=0x7f3a1410c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 --2- 172.21.15.89:0/3446666430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a14103dc0 0x7f3a1410c2b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a040098f0 tx=0x7f3a04009c00 comp rx=0 tx=0).stop 2024-03-21T13:12:11.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 -- 172.21.15.89:0/3446666430 shutdown_connections 2024-03-21T13:12:11.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 --2- 172.21.15.89:0/3446666430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a1410c880 0x7f3a1410ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 --2- 172.21.15.89:0/3446666430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a141034a0 0x7f3a14103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 --2- 172.21.15.89:0/3446666430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a14103dc0 0x7f3a1410c2b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 -- 172.21.15.89:0/3446666430 >> 172.21.15.89:0/3446666430 conn(0x7f3a140781c0 msgr2=0x7f3a140785d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 -- 172.21.15.89:0/3446666430 shutdown_connections 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.778+0000 7f3a191a0700 1 -- 172.21.15.89:0/3446666430 wait complete. 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 Processor -- start 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 -- start start 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a141034a0 0x7f3a14197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a14103dc0 0x7f3a14198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a1410c880 0x7f3a1419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14111a30 con 0x7f3a14103dc0 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a141118b0 con 0x7f3a1410c880 2024-03-21T13:12:11.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a191a0700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14111bb0 con 0x7f3a141034a0 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a1259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a14103dc0 0x7f3a14198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a12d9d700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a141034a0 0x7f3a14197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a1259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a14103dc0 0x7f3a14198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53546/0 (socket says 172.21.15.89:53546) 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a12d9d700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a141034a0 0x7f3a14197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:51464/0 (socket says 172.21.15.89:51464) 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.779+0000 7f3a1259c700 1 -- 172.21.15.89:0/1275738467 learned_addr learned my addr 172.21.15.89:0/1275738467 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a1359e700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a1410c880 0x7f3a1419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a1259c700 1 -- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a141034a0 msgr2=0x7f3a14197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a1259c700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a141034a0 0x7f3a14197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a1259c700 1 -- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a1410c880 msgr2=0x7f3a1419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a1259c700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a1410c880 0x7f3a1419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a1259c700 1 -- 172.21.15.89:0/1275738467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1419cfe0 con 0x7f3a14103dc0 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a12d9d700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3a141034a0 0x7f3a14197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:11.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a1259c700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a14103dc0 0x7f3a14198510 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a04009230 tx=0x7f3a0400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:11.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0400b660 con 0x7f3a14103dc0 2024-03-21T13:12:11.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1419d270 con 0x7f3a14103dc0 2024-03-21T13:12:11.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3a04005040 con 0x7f3a14103dc0 2024-03-21T13:12:11.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a040038e0 con 0x7f3a14103dc0 2024-03-21T13:12:11.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.780+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a141a4b50 con 0x7f3a14103dc0 2024-03-21T13:12:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.781+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a14062860 con 0x7f3a14103dc0 2024-03-21T13:12:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f3a0400b800 con 0x7f3a14103dc0 2024-03-21T13:12:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f39fbfff700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f39fc06e340 0x7f39fc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f3a0408f900 con 0x7f3a14103dc0 2024-03-21T13:12:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f3a12d9d700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f39fc06e340 0x7f39fc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f39fbfff700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] conn(0x7f39fc0781d0 0x7f39fc07a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 --> [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a0400bd80 con 0x7f39fc0781d0 2024-03-21T13:12:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a040942c0 con 0x7f3a14103dc0 2024-03-21T13:12:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f3a12d9d700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f39fc06e340 0x7f39fc070800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3a08005950 tx=0x7f3a0800a300 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.782+0000 7f3a1359e700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] conn(0x7f39fc0781d0 0x7f39fc07a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.783+0000 7f3a1359e700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] conn(0x7f39fc0781d0 0x7f39fc07a670 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.784+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== osd.5 v2:172.21.15.152:6808/3145783803 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a0400bd80 con 0x7f39fc0781d0 2024-03-21T13:12:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.799+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 --> [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a14062c60 con 0x7f39fc0781d0 2024-03-21T13:12:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f39fbfff700 1 -- 172.21.15.89:0/1275738467 <== osd.5 v2:172.21.15.152:6808/3145783803 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3a14062c60 con 0x7f39fc0781d0 2024-03-21T13:12:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] conn(0x7f39fc0781d0 msgr2=0x7f39fc07a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.152:6808/3145783803,v1:172.21.15.152:6809/3145783803] conn(0x7f39fc0781d0 0x7f39fc07a670 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f39fc06e340 msgr2=0x7f39fc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f39fc06e340 0x7f39fc070800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3a08005950 tx=0x7f3a0800a300 comp rx=0 tx=0).stop 2024-03-21T13:12:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a14103dc0 msgr2=0x7f3a14198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 --2- 172.21.15.89:0/1275738467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a14103dc0 0x7f3a14198510 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a04009230 tx=0x7f3a0400fa00 comp rx=0 tx=0).stop 2024-03-21T13:12:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a1359e700 1 -- 172.21.15.89:0/1275738467 reap_dead start 2024-03-21T13:12:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 shutdown_connections 2024-03-21T13:12:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.800+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 >> 172.21.15.89:0/1275738467 conn(0x7f3a140781c0 msgr2=0x7f3a14104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.801+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 shutdown_connections 2024-03-21T13:12:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.801+0000 7f3a191a0700 1 -- 172.21.15.89:0/1275738467 wait complete. 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 -- 172.21.15.89:0/1942932916 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bd406ae70 msgr2=0x7f0bd406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 --2- 172.21.15.89:0/1942932916 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bd406ae70 0x7f0bd406b2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40098f0 tx=0x7f0bc4009c00 comp rx=0 tx=0).stop 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 -- 172.21.15.89:0/1942932916 shutdown_connections 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 --2- 172.21.15.89:0/1942932916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd4104e10 0x7f0bd4105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 --2- 172.21.15.89:0/1942932916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bd41072a0 0x7f0bd406a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 --2- 172.21.15.89:0/1942932916 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bd406ae70 0x7f0bd406b2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 -- 172.21.15.89:0/1942932916 >> 172.21.15.89:0/1942932916 conn(0x7f0bd40770e0 msgr2=0x7f0bd40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 -- 172.21.15.89:0/1942932916 shutdown_connections 2024-03-21T13:12:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.657+0000 7f0bdb39b700 1 -- 172.21.15.89:0/1942932916 wait complete. 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 Processor -- start 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 -- start start 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd406ae70 0x7f0bd419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bd4104e10 0x7f0bd419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bd41072a0 0x7f0bd41a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4118150 con 0x7f0bd406ae70 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4117fd0 con 0x7f0bd4104e10 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bdb39b700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd41182d0 con 0x7f0bd41072a0 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd406ae70 0x7f0bd419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd406ae70 0x7f0bd419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53530/0 (socket says 172.21.15.89:53530) 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 -- 172.21.15.89:0/3620350093 learned_addr learned my addr 172.21.15.89:0/3620350093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9938700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bd41072a0 0x7f0bd41a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd8936700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bd4104e10 0x7f0bd419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 -- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bd41072a0 msgr2=0x7f0bd41a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bd41072a0 0x7f0bd41a2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 -- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bd4104e10 msgr2=0x7f0bd419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bd4104e10 0x7f0bd419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 -- 172.21.15.89:0/3620350093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd41a3650 con 0x7f0bd406ae70 2024-03-21T13:12:11.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd8936700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bd4104e10 0x7f0bd419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T13:12:11.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bd9137700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd406ae70 0x7f0bd419e640 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c9b0 tx=0x7f0bd000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.658+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0007b00 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.659+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0bd0007ca0 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.659+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd000fe60 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.659+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd41a3940 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.659+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd41ab1c0 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.659+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0bd4100b00 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.660+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f0bd000f450 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.660+0000 7f0bca7fc700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0bc006e340 0x7f0bc0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.660+0000 7f0bd8936700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0bc006e340 0x7f0bc0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.660+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f0bd008e810 con 0x7f0bd406ae70 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.661+0000 7f0bca7fc700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] conn(0x7f0bc0078230 0x7f0bc007a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.661+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 --> [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0bc007ada0 con 0x7f0bc0078230 2024-03-21T13:12:11.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.661+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bd00932c0 con 0x7f0bd406ae70 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.661+0000 7f0bd8936700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0bc006e340 0x7f0bc0070800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009230 tx=0x7f0bc400fd60 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.661+0000 7f0bd9938700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] conn(0x7f0bc0078230 0x7f0bc007a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.661+0000 7f0bd9938700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] conn(0x7f0bc0078230 0x7f0bc007a6d0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.662+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== osd.6 v2:172.21.15.152:6816/3295577655 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0bc007ada0 con 0x7f0bc0078230 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.679+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 --> [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0bd4101710 con 0x7f0bc0078230 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.679+0000 7f0bca7fc700 1 -- 172.21.15.89:0/3620350093 <== osd.6 v2:172.21.15.152:6816/3295577655 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0bd4101710 con 0x7f0bc0078230 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] conn(0x7f0bc0078230 msgr2=0x7f0bc007a6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.152:6816/3295577655,v1:172.21.15.152:6817/3295577655] conn(0x7f0bc0078230 0x7f0bc007a6d0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0bc006e340 msgr2=0x7f0bc0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0bc006e340 0x7f0bc0070800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009230 tx=0x7f0bc400fd60 comp rx=0 tx=0).stop 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd406ae70 msgr2=0x7f0bd419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 --2- 172.21.15.89:0/3620350093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd406ae70 0x7f0bd419e640 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c9b0 tx=0x7f0bd000ccc0 comp rx=0 tx=0).stop 2024-03-21T13:12:11.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bd9938700 1 -- 172.21.15.89:0/3620350093 reap_dead start 2024-03-21T13:12:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 shutdown_connections 2024-03-21T13:12:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 >> 172.21.15.89:0/3620350093 conn(0x7f0bd40770e0 msgr2=0x7f0bd4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 shutdown_connections 2024-03-21T13:12:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:11.680+0000 7f0bdb39b700 1 -- 172.21.15.89:0/3620350093 wait complete. 2024-03-21T13:12:12.941 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:12 smithi089 ceph-mon[30095]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:12.942 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:12 smithi089 ceph-mon[39209]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:12 smithi152 ceph-mon[34051]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:13.235 INFO:teuthology.orchestra.run.smithi089.stdout:111669149731 2024-03-21T13:12:13.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T13:12:13.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 -- 172.21.15.89:0/1861423374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a8101370 msgr2=0x7f21a81017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 --2- 172.21.15.89:0/1861423374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a8101370 0x7f21a81017f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f21980098f0 tx=0x7f2198009c00 comp rx=0 tx=0).stop 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 -- 172.21.15.89:0/1861423374 shutdown_connections 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 --2- 172.21.15.89:0/1861423374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21a8101d30 0x7f21a810df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 --2- 172.21.15.89:0/1861423374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a81071c0 0x7f21a81075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 --2- 172.21.15.89:0/1861423374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a8101370 0x7f21a81017f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 -- 172.21.15.89:0/1861423374 >> 172.21.15.89:0/1861423374 conn(0x7f21a8077f50 msgr2=0x7f21a8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 -- 172.21.15.89:0/1861423374 shutdown_connections 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 -- 172.21.15.89:0/1861423374 wait complete. 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 Processor -- start 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.950+0000 7f21af6cf700 1 -- start start 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a8101370 0x7f21a8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21a8101d30 0x7f21a8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:13.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a81071c0 0x7f21a819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8110860 con 0x7f21a81071c0 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81106e0 con 0x7f21a8101d30 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81109e0 con 0x7f21a8101370 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a81071c0 0x7f21a819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a81071c0 0x7f21a819c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53552/0 (socket says 172.21.15.89:53552) 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21ad46b700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a8101370 0x7f21a8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 -- 172.21.15.89:0/4264515710 learned_addr learned my addr 172.21.15.89:0/4264515710 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21acc6a700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21a8101d30 0x7f21a8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:13.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 -- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a8101370 msgr2=0x7f21a8197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a8101370 0x7f21a8197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 -- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21a8101d30 msgr2=0x7f21a8198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21a8101d30 0x7f21a8198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 -- 172.21.15.89:0/4264515710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a819ce60 con 0x7f21a81071c0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21adc6c700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a81071c0 0x7f21a819c720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21a000efc0 tx=0x7f21a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a00092e0 con 0x7f21a81071c0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a819d0f0 con 0x7f21a81071c0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f21a0004bb0 con 0x7f21a81071c0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a000ce60 con 0x7f21a81071c0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.951+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a81a49d0 con 0x7f21a81071c0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.952+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21a806b210 con 0x7f21a81071c0 2024-03-21T13:12:13.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.953+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f21a001b030 con 0x7f21a81071c0 2024-03-21T13:12:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.953+0000 7f219e7fc700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f219406e340 0x7f2194070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.953+0000 7f21ad46b700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f219406e340 0x7f2194070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.953+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f21a008e290 con 0x7f21a81071c0 2024-03-21T13:12:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.953+0000 7f219e7fc700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] conn(0x7f21940781d0 0x7f219407a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.953+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 --> [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f219407ad40 con 0x7f21940781d0 2024-03-21T13:12:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.953+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21a0095050 con 0x7f21a81071c0 2024-03-21T13:12:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.954+0000 7f21ad46b700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f219406e340 0x7f2194070800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009750 tx=0x7f21a4009310 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.954+0000 7f21acc6a700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] conn(0x7f21940781d0 0x7f219407a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.954+0000 7f21acc6a700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] conn(0x7f21940781d0 0x7f219407a670 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.955+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== osd.4 v2:172.21.15.152:6800/2252846533 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f219407ad40 con 0x7f21940781d0 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.971+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 --> [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21a81027c0 con 0x7f21940781d0 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f219e7fc700 1 -- 172.21.15.89:0/4264515710 <== osd.4 v2:172.21.15.152:6800/2252846533 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f21a81027c0 con 0x7f21940781d0 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] conn(0x7f21940781d0 msgr2=0x7f219407a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.152:6800/2252846533,v1:172.21.15.152:6801/2252846533] conn(0x7f21940781d0 0x7f219407a670 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f219406e340 msgr2=0x7f2194070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f219406e340 0x7f2194070800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009750 tx=0x7f21a4009310 comp rx=0 tx=0).stop 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a81071c0 msgr2=0x7f21a819c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 --2- 172.21.15.89:0/4264515710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a81071c0 0x7f21a819c720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21a000efc0 tx=0x7f21a000c3a0 comp rx=0 tx=0).stop 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21adc6c700 1 -- 172.21.15.89:0/4264515710 reap_dead start 2024-03-21T13:12:13.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 shutdown_connections 2024-03-21T13:12:13.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 >> 172.21.15.89:0/4264515710 conn(0x7f21a8077f50 msgr2=0x7f21a806a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:13.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 shutdown_connections 2024-03-21T13:12:13.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:12.972+0000 7f21af6cf700 1 -- 172.21.15.89:0/4264515710 wait complete. 2024-03-21T13:12:13.979 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:14.883 INFO:teuthology.orchestra.run.smithi089.stdout:137438953503 2024-03-21T13:12:14.932 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T13:12:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:14 smithi152 ceph-mon[34051]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:14 smithi089 ceph-mon[30095]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:15.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:14 smithi089 ceph-mon[39209]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 -- 172.21.15.89:0/3098209723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 msgr2=0x7f41841086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 --2- 172.21.15.89:0/3098209723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 0x7f41841086d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4178009990 tx=0x7f4178009ca0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 -- 172.21.15.89:0/3098209723 shutdown_connections 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 --2- 172.21.15.89:0/3098209723 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4184102bd0 0x7f418410f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 --2- 172.21.15.89:0/3098209723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 0x7f41841086d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 --2- 172.21.15.89:0/3098209723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4184102210 0x7f4184102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 -- 172.21.15.89:0/3098209723 >> 172.21.15.89:0/3098209723 conn(0x7f4184079250 msgr2=0x7f4184079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 -- 172.21.15.89:0/3098209723 shutdown_connections 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.241+0000 7f418ac7d700 1 -- 172.21.15.89:0/3098209723 wait complete. 2024-03-21T13:12:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 Processor -- start 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 -- start start 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4184102210 0x7f4184074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4184102bd0 0x7f41840749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 0x7f4184074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41841118c0 con 0x7f41841082f0 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4184111740 con 0x7f4184102210 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418ac7d700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4184111a40 con 0x7f4184102bd0 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418921a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 0x7f4184074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418921a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 0x7f4184074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53568/0 (socket says 172.21.15.89:53568) 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f4188a19700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4184102210 0x7f4184074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.242+0000 7f418921a700 1 -- 172.21.15.89:0/1521242375 learned_addr learned my addr 172.21.15.89:0/1521242375 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f4183fff700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4184102bd0 0x7f41840749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418921a700 1 -- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4184102bd0 msgr2=0x7f41840749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418921a700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4184102bd0 0x7f41840749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418921a700 1 -- 172.21.15.89:0/1521242375 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4184102210 msgr2=0x7f4184074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418921a700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4184102210 0x7f4184074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418921a700 1 -- 172.21.15.89:0/1521242375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41841a62f0 con 0x7f41841082f0 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f4183fff700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4184102bd0 0x7f41840749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418921a700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 0x7f4184074f10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f417c00ef30 tx=0x7f417c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0092e0 con 0x7f41841082f0 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f417c004bb0 con 0x7f41841082f0 2024-03-21T13:12:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41841a6580 con 0x7f41841082f0 2024-03-21T13:12:15.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c004020 con 0x7f41841082f0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.243+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41841a6ac0 con 0x7f41841082f0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.244+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f41841077c0 con 0x7f41841082f0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.245+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f417c01b030 con 0x7f41841082f0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.245+0000 7f4181ffb700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f417406e340 0x7f4174070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.246+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f417c08e3b0 con 0x7f41841082f0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.246+0000 7f4188a19700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f417406e340 0x7f4174070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.246+0000 7f4181ffb700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] conn(0x7f41740781d0 0x7f417407a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.246+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 --> [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f417407ad40 con 0x7f41740781d0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.246+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f417c095050 con 0x7f41841082f0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.246+0000 7f4188a19700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f417406e340 0x7f4174070800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f417800b2e0 tx=0x7f4178005960 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.246+0000 7f4183fff700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] conn(0x7f41740781d0 0x7f417407a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.247+0000 7f4183fff700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] conn(0x7f41740781d0 0x7f417407a670 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.247+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== osd.7 v2:172.21.15.152:6824/440724144 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f417407ad40 con 0x7f41740781d0 2024-03-21T13:12:15.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.279+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 --> [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f41841038d0 con 0x7f41740781d0 2024-03-21T13:12:15.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.279+0000 7f4181ffb700 1 -- 172.21.15.89:0/1521242375 <== osd.7 v2:172.21.15.152:6824/440724144 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f41841038d0 con 0x7f41740781d0 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 >> [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] conn(0x7f41740781d0 msgr2=0x7f417407a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.152:6824/440724144,v1:172.21.15.152:6825/440724144] conn(0x7f41740781d0 0x7f417407a670 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f417406e340 msgr2=0x7f4174070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f417406e340 0x7f4174070800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f417800b2e0 tx=0x7f4178005960 comp rx=0 tx=0).stop 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 msgr2=0x7f4184074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 --2- 172.21.15.89:0/1521242375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41841082f0 0x7f4184074f10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f417c00ef30 tx=0x7f417c00c3a0 comp rx=0 tx=0).stop 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418921a700 1 -- 172.21.15.89:0/1521242375 reap_dead start 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 shutdown_connections 2024-03-21T13:12:15.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 >> 172.21.15.89:0/1521242375 conn(0x7f4184079250 msgr2=0x7f4184105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:15.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 shutdown_connections 2024-03-21T13:12:15.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:15.280+0000 7f418ac7d700 1 -- 172.21.15.89:0/1521242375 wait complete. 2024-03-21T13:12:16.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:15 smithi152 ceph-mon[34051]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:16.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:15 smithi089 ceph-mon[39209]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:15 smithi089 ceph-mon[30095]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:17.630 INFO:teuthology.orchestra.run.smithi089.stdout:206158430229 2024-03-21T13:12:17.631 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T13:12:17.631 INFO:teuthology.orchestra.run.smithi089.stdout:171798691866 2024-03-21T13:12:17.632 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T13:12:17.767 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:12:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:17.768 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:17 smithi089 ceph-mon[30095]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:17.921 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:17.922 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:17.967 INFO:teuthology.orchestra.run.smithi089.stdout:240518168592 2024-03-21T13:12:17.968 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T13:12:17.974 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:17.984 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:18.066 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:17 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:18.066 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:17 smithi089 ceph-mon[39209]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:18.066 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:17 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:18.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:17 smithi152 ceph-mon[34051]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:18.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:17 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:18.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:12:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:12:20.016 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:19 smithi089 ceph-mon[30095]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:20.016 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:19 smithi089 ceph-mon[39209]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:19 smithi152 ceph-mon[34051]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:22.030 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:21 smithi089 ceph-mon[30095]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:22.031 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:21 smithi089 ceph-mon[39209]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:21 smithi152 ceph-mon[34051]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:23.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:23 smithi089 ceph-mon[30095]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:23.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:23 smithi089 ceph-mon[39209]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:23 smithi152 ceph-mon[34051]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:25.667 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:25.667 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.864+0000 7f485cf6e700 1 -- 172.21.15.89:0/87668888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806b4c0 msgr2=0x7f4858106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.864+0000 7f485cf6e700 1 --2- 172.21.15.89:0/87668888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806b4c0 0x7f4858106df0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f484c009920 tx=0x7f484c009c30 comp rx=0 tx=0).stop 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 -- 172.21.15.89:0/87668888 shutdown_connections 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 --2- 172.21.15.89:0/87668888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4858107520 0x7f485810f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 --2- 172.21.15.89:0/87668888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f485806ab10 0x7f485806aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 --2- 172.21.15.89:0/87668888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806b4c0 0x7f4858106df0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 -- 172.21.15.89:0/87668888 >> 172.21.15.89:0/87668888 conn(0x7f4858077f50 msgr2=0x7f4858078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 -- 172.21.15.89:0/87668888 shutdown_connections 2024-03-21T13:12:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 -- 172.21.15.89:0/87668888 wait complete. 2024-03-21T13:12:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.865+0000 7f485cf6e700 1 Processor -- start 2024-03-21T13:12:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485cf6e700 1 -- start start 2024-03-21T13:12:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485cf6e700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806ab10 0x7f4858197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485cf6e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f485806b4c0 0x7f4858198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485cf6e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4858107520 0x7f485819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485cf6e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4858111960 con 0x7f4858107520 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485cf6e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48581117e0 con 0x7f485806b4c0 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485cf6e700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4858111ae0 con 0x7f485806ab10 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806ab10 0x7f4858197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f4856d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4858107520 0x7f485819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806ab10 0x7f4858197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:38338/0 (socket says 172.21.15.89:38338) 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 -- 172.21.15.89:0/3723714391 learned_addr learned my addr 172.21.15.89:0/3723714391 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f4855d9b700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f485806b4c0 0x7f4858198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 -- 172.21.15.89:0/3723714391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f485806b4c0 msgr2=0x7f4858198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f485806b4c0 0x7f4858198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 -- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4858107520 msgr2=0x7f485819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4858107520 0x7f485819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 -- 172.21.15.89:0/3723714391 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485819cef0 con 0x7f485806ab10 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f4856d9d700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4858107520 0x7f485819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f4855d9b700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f485806b4c0 0x7f4858198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.866+0000 7f485659c700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806ab10 0x7f4858197ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f484800c9b0 tx=0x7f484800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.867+0000 7f48477fe700 1 -- 172.21.15.89:0/3723714391 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848007b00 con 0x7f485806ab10 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.867+0000 7f48477fe700 1 -- 172.21.15.89:0/3723714391 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4848007ca0 con 0x7f485806ab10 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.867+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485819d1e0 con 0x7f485806ab10 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.867+0000 7f48477fe700 1 -- 172.21.15.89:0/3723714391 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484800fe60 con 0x7f485806ab10 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.867+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48581a4a60 con 0x7f485806ab10 2024-03-21T13:12:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.868+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48581038d0 con 0x7f485806ab10 2024-03-21T13:12:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.868+0000 7f48477fe700 1 -- 172.21.15.89:0/3723714391 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f484800f450 con 0x7f485806ab10 2024-03-21T13:12:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.869+0000 7f48477fe700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f484006e340 0x7f4840070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.869+0000 7f48477fe700 1 -- 172.21.15.89:0/3723714391 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f484808e7e0 con 0x7f485806ab10 2024-03-21T13:12:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.869+0000 7f4855d9b700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f484006e340 0x7f4840070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.869+0000 7f4855d9b700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f484006e340 0x7f4840070800 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f484c0098f0 tx=0x7f484c019040 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:25.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:25.874+0000 7f48477fe700 1 -- 172.21.15.89:0/3723714391 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4848014030 con 0x7f485806ab10 2024-03-21T13:12:25.969 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:25 smithi089 ceph-mon[30095]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:25.969 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:25 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:25.970 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:25 smithi089 ceph-mon[39209]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:25.970 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:25 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:25 smithi152 ceph-mon[34051]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:25 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.086+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4858199240 con 0x7f485806ab10 2024-03-21T13:12:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.086+0000 7f48477fe700 1 -- 172.21.15.89:0/3723714391 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f484805bad0 con 0x7f485806ab10 2024-03-21T13:12:26.087 INFO:teuthology.orchestra.run.smithi089.stdout:55834574896 2024-03-21T13:12:26.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f484006e340 msgr2=0x7f4840070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:26.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f484006e340 0x7f4840070800 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f484c0098f0 tx=0x7f484c019040 comp rx=0 tx=0).stop 2024-03-21T13:12:26.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806ab10 msgr2=0x7f4858197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:26.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806ab10 0x7f4858197ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f484800c9b0 tx=0x7f484800ccc0 comp rx=0 tx=0).stop 2024-03-21T13:12:26.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 shutdown_connections 2024-03-21T13:12:26.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4858107520 0x7f485819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:26.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f484006e340 0x7f4840070800 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:26.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f485806ab10 0x7f4858197ee0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:26.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 --2- 172.21.15.89:0/3723714391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f485806b4c0 0x7f4858198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:26.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 >> 172.21.15.89:0/3723714391 conn(0x7f4858077f50 msgr2=0x7f48580ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:26.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.088+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 shutdown_connections 2024-03-21T13:12:26.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:26.089+0000 7f485cf6e700 1 -- 172.21.15.89:0/3723714391 wait complete. 2024-03-21T13:12:27.064 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3723714391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T13:12:27.064 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3723714391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T13:12:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3723714391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 -- 172.21.15.89:0/902832180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 msgr2=0x7fa4f010df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 --2- 172.21.15.89:0/902832180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 0x7fa4f010df70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009f90 tx=0x7fa4e800b330 comp rx=0 tx=0).stop 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 -- 172.21.15.89:0/902832180 shutdown_connections 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 --2- 172.21.15.89:0/902832180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 0x7fa4f010df70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 --2- 172.21.15.89:0/902832180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4f01060d0 0x7fa4f01064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 --2- 172.21.15.89:0/902832180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 0x7fa4f0102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 -- 172.21.15.89:0/902832180 >> 172.21.15.89:0/902832180 conn(0x7fa4f00763d0 msgr2=0x7fa4f00767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 -- 172.21.15.89:0/902832180 shutdown_connections 2024-03-21T13:12:27.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 -- 172.21.15.89:0/902832180 wait complete. 2024-03-21T13:12:27.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 Processor -- start 2024-03-21T13:12:27.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.257+0000 7fa4f6654700 1 -- start start 2024-03-21T13:12:27.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f6654700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 0x7fa4f0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f6654700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 0x7fa4f0199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f6654700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4f01060d0 0x7fa4f01a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f6654700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0110880 con 0x7fa4f01025a0 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f6654700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0110700 con 0x7fa4f01060d0 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f6654700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0110a00 con 0x7fa4f0101be0 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 0x7fa4f0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 0x7fa4f0198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:38360/0 (socket says 172.21.15.89:38360) 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 -- 172.21.15.89:0/3175951418 learned_addr learned my addr 172.21.15.89:0/3175951418 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4ef7fe700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 0x7fa4f0199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f4bf1700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4f01060d0 0x7fa4f01a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 -- 172.21.15.89:0/3175951418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4f01060d0 msgr2=0x7fa4f01a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:27.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4f01060d0 0x7fa4f01a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 -- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 msgr2=0x7fa4f0199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 0x7fa4f0199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 -- 172.21.15.89:0/3175951418 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f01a4210 con 0x7fa4f0101be0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4f4bf1700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4f01060d0 0x7fa4f01a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4ef7fe700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 0x7fa4f0199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.258+0000 7fa4effff700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 0x7fa4f0198c50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d950 tx=0x7fa4e400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.259+0000 7fa4ed7fa700 1 -- 172.21.15.89:0/3175951418 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4016070 con 0x7fa4f0101be0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.259+0000 7fa4ed7fa700 1 -- 172.21.15.89:0/3175951418 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa4e4009dc0 con 0x7fa4f0101be0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.259+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f01a4440 con 0x7fa4f0101be0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.259+0000 7fa4ed7fa700 1 -- 172.21.15.89:0/3175951418 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400f3c0 con 0x7fa4f0101be0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.259+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f01a49d0 con 0x7fa4f0101be0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.260+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f0068e50 con 0x7fa4f0101be0 2024-03-21T13:12:27.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.260+0000 7fa4ed7fa700 1 -- 172.21.15.89:0/3175951418 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fa4e4019070 con 0x7fa4f0101be0 2024-03-21T13:12:27.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.260+0000 7fa4ed7fa700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fa4d806e340 0x7fa4d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:27.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.261+0000 7fa4ef7fe700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fa4d806e340 0x7fa4d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:27.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.261+0000 7fa4ed7fa700 1 -- 172.21.15.89:0/3175951418 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fa4e408e320 con 0x7fa4f0101be0 2024-03-21T13:12:27.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.261+0000 7fa4ef7fe700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fa4d806e340 0x7fa4d8070800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009750 tx=0x7fa4e0009310 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:27.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.265+0000 7fa4ed7fa700 1 -- 172.21.15.89:0/3175951418 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa4e405b610 con 0x7fa4f0101be0 2024-03-21T13:12:27.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.466+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa4f01a4c30 con 0x7fa4f0101be0 2024-03-21T13:12:27.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.466+0000 7fa4ed7fa700 1 -- 172.21.15.89:0/3175951418 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa4e405f060 con 0x7fa4f0101be0 2024-03-21T13:12:27.467 INFO:teuthology.orchestra.run.smithi089.stdout:111669149735 2024-03-21T13:12:27.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fa4d806e340 msgr2=0x7fa4d8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:27.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fa4d806e340 0x7fa4d8070800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009750 tx=0x7fa4e0009310 comp rx=0 tx=0).stop 2024-03-21T13:12:27.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 msgr2=0x7fa4f0198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:27.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 0x7fa4f0198c50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d950 tx=0x7fa4e400dc60 comp rx=0 tx=0).stop 2024-03-21T13:12:27.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 shutdown_connections 2024-03-21T13:12:27.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4f01060d0 0x7fa4f01a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fa4d806e340 0x7fa4d8070800 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa4f0101be0 0x7fa4f0198c50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 --2- 172.21.15.89:0/3175951418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f01025a0 0x7fa4f0199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:27.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.468+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 >> 172.21.15.89:0/3175951418 conn(0x7fa4f00763d0 msgr2=0x7fa4f0104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:27.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.469+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 shutdown_connections 2024-03-21T13:12:27.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:27.469+0000 7fa4f6654700 1 -- 172.21.15.89:0/3175951418 wait complete. 2024-03-21T13:12:27.794 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:12:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:28.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:27 smithi089 ceph-mon[30095]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:28.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3175951418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T13:12:28.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:27 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:28.048 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:27 smithi089 ceph-mon[39209]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:28.048 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3175951418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T13:12:28.048 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:27 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:27 smithi152 ceph-mon[34051]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3175951418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T13:12:28.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:27 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:28.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:12:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 -- 172.21.15.89:0/977573387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 msgr2=0x7f7c2810df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 --2- 172.21.15.89:0/977573387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 0x7f7c2810df40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c200099c0 tx=0x7f7c20009cd0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 -- 172.21.15.89:0/977573387 shutdown_connections 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 --2- 172.21.15.89:0/977573387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 0x7f7c2810df40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 --2- 172.21.15.89:0/977573387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c28101320 0x7f7c28101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 --2- 172.21.15.89:0/977573387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c28101cd0 0x7f7c28105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 -- 172.21.15.89:0/977573387 >> 172.21.15.89:0/977573387 conn(0x7f7c280770e0 msgr2=0x7f7c280774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 -- 172.21.15.89:0/977573387 shutdown_connections 2024-03-21T13:12:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.694+0000 7f7c302e7700 1 -- 172.21.15.89:0/977573387 wait complete. 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 Processor -- start 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 -- start start 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c28101320 0x7f7c28102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c28101cd0 0x7f7c28103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 0x7f7c281037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c28110840 con 0x7f7c28101320 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281106c0 con 0x7f7c28101cd0 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c302e7700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281109c0 con 0x7f7c28106370 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e083700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c28101320 0x7f7c28102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e884700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 0x7f7c281037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e083700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c28101320 0x7f7c28102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46832/0 (socket says 172.21.15.89:46832) 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e083700 1 -- 172.21.15.89:0/2642598297 learned_addr learned my addr 172.21.15.89:0/2642598297 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2d882700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c28101cd0 0x7f7c28103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e884700 1 -- 172.21.15.89:0/2642598297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c28101cd0 msgr2=0x7f7c28103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e884700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c28101cd0 0x7f7c28103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e884700 1 -- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c28101320 msgr2=0x7f7c28102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e884700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c28101320 0x7f7c28102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.695+0000 7f7c2e884700 1 -- 172.21.15.89:0/2642598297 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c281a63c0 con 0x7f7c28106370 2024-03-21T13:12:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.696+0000 7f7c2e884700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 0x7f7c281037b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7c200098e0 tx=0x7f7c200045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:28.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.696+0000 7f7c1f7fe700 1 -- 172.21.15.89:0/2642598297 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20009e60 con 0x7f7c28106370 2024-03-21T13:12:28.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.696+0000 7f7c1f7fe700 1 -- 172.21.15.89:0/2642598297 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7c20003e50 con 0x7f7c28106370 2024-03-21T13:12:28.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.696+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c281a6650 con 0x7f7c28106370 2024-03-21T13:12:28.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.696+0000 7f7c1f7fe700 1 -- 172.21.15.89:0/2642598297 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20005030 con 0x7f7c28106370 2024-03-21T13:12:28.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.696+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c281a6aa0 con 0x7f7c28106370 2024-03-21T13:12:28.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.697+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c28068e50 con 0x7f7c28106370 2024-03-21T13:12:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.697+0000 7f7c1f7fe700 1 -- 172.21.15.89:0/2642598297 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f7c200039d0 con 0x7f7c28106370 2024-03-21T13:12:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.698+0000 7f7c1f7fe700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7c1406e340 0x7f7c14070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.698+0000 7f7c2e083700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7c1406e340 0x7f7c14070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.698+0000 7f7c1f7fe700 1 -- 172.21.15.89:0/2642598297 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f7c2008f390 con 0x7f7c28106370 2024-03-21T13:12:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.698+0000 7f7c2e083700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7c1406e340 0x7f7c14070800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400e250 tx=0x7f7c2400b040 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:28.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.702+0000 7f7c1f7fe700 1 -- 172.21.15.89:0/2642598297 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c20013070 con 0x7f7c28106370 2024-03-21T13:12:28.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.902+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7c281a6dc0 con 0x7f7c28106370 2024-03-21T13:12:28.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.903+0000 7f7c1f7fe700 1 -- 172.21.15.89:0/2642598297 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7c2005c680 con 0x7f7c28106370 2024-03-21T13:12:28.903 INFO:teuthology.orchestra.run.smithi089.stdout:137438953506 2024-03-21T13:12:28.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7c1406e340 msgr2=0x7f7c14070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:28.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7c1406e340 0x7f7c14070800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400e250 tx=0x7f7c2400b040 comp rx=0 tx=0).stop 2024-03-21T13:12:28.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 msgr2=0x7f7c281037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 0x7f7c281037b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7c200098e0 tx=0x7f7c200045e0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 shutdown_connections 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f7c1406e340 0x7f7c14070800 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c28106370 0x7f7c281037b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c28101320 0x7f7c28102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 --2- 172.21.15.89:0/2642598297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c28101cd0 0x7f7c28103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 >> 172.21.15.89:0/2642598297 conn(0x7f7c280770e0 msgr2=0x7f7c28105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 shutdown_connections 2024-03-21T13:12:28.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:28.905+0000 7f7c302e7700 1 -- 172.21.15.89:0/2642598297 wait complete. 2024-03-21T13:12:29.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 -- 172.21.15.89:0/1184364097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc108260 msgr2=0x7f41cc108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 --2- 172.21.15.89:0/1184364097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc108260 0x7f41cc108640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009920 tx=0x7f41c8009c30 comp rx=0 tx=0).stop 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 -- 172.21.15.89:0/1184364097 shutdown_connections 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 --2- 172.21.15.89:0/1184364097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc102bd0 0x7f41cc10f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 --2- 172.21.15.89:0/1184364097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc108260 0x7f41cc108640 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 --2- 172.21.15.89:0/1184364097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 0x7f41cc102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 -- 172.21.15.89:0/1184364097 >> 172.21.15.89:0/1184364097 conn(0x7f41cc0792e0 msgr2=0x7f41cc0796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 -- 172.21.15.89:0/1184364097 shutdown_connections 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 -- 172.21.15.89:0/1184364097 wait complete. 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 Processor -- start 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.455+0000 7f41d39f1700 1 -- start start 2024-03-21T13:12:29.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d39f1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 0x7f41cc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d39f1700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc102bd0 0x7f41cc1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d39f1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc108260 0x7f41cc19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d39f1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc1118f0 con 0x7f41cc108260 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d39f1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc111770 con 0x7f41cc102210 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d39f1700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc111a70 con 0x7f41cc102bd0 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc102bd0 0x7f41cc1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d178d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 0x7f41cc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc102bd0 0x7f41cc1984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:48296/0 (socket says 172.21.15.89:48296) 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d178d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 0x7f41cc197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.89:49806/0 (socket says 172.21.15.89:49806) 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 -- 172.21.15.89:0/3909874801 learned_addr learned my addr 172.21.15.89:0/3909874801 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d1f8e700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc108260 0x7f41cc19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 -- 172.21.15.89:0/3909874801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 msgr2=0x7f41cc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 0x7f41cc197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 -- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc108260 msgr2=0x7f41cc19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc108260 0x7f41cc19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 -- 172.21.15.89:0/3909874801 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc19cf90 con 0x7f41cc102bd0 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d178d700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 0x7f41cc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d1f8e700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc108260 0x7f41cc19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.456+0000 7f41d0f8c700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc102bd0 0x7f41cc1984c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009f70 tx=0x7f41bc0074a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.457+0000 7f41c27fc700 1 -- 172.21.15.89:0/3909874801 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc021070 con 0x7f41cc102bd0 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.457+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41cc19d280 con 0x7f41cc102bd0 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.457+0000 7f41c27fc700 1 -- 172.21.15.89:0/3909874801 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f41bc007d90 con 0x7f41cc102bd0 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.457+0000 7f41c27fc700 1 -- 172.21.15.89:0/3909874801 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc013980 con 0x7f41cc102bd0 2024-03-21T13:12:29.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.457+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41cc0743c0 con 0x7f41cc102bd0 2024-03-21T13:12:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.458+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41cc068e50 con 0x7f41cc102bd0 2024-03-21T13:12:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.459+0000 7f41c27fc700 1 -- 172.21.15.89:0/3909874801 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f41bc024070 con 0x7f41cc102bd0 2024-03-21T13:12:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.459+0000 7f41c27fc700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f41b806e4e0 0x7f41b80709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.459+0000 7f41c27fc700 1 -- 172.21.15.89:0/3909874801 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f41bc095e00 con 0x7f41cc102bd0 2024-03-21T13:12:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.459+0000 7f41d178d700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f41b806e4e0 0x7f41b80709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.460+0000 7f41d178d700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f41b806e4e0 0x7f41b80709a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009810 tx=0x7f41c8019040 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:29.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.464+0000 7f41c27fc700 1 -- 172.21.15.89:0/3909874801 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41bc066b40 con 0x7f41cc102bd0 2024-03-21T13:12:29.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.706+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f41cc074680 con 0x7f41cc102bd0 2024-03-21T13:12:29.708 INFO:teuthology.orchestra.run.smithi089.stdout:77309411372 2024-03-21T13:12:29.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.706+0000 7f41c27fc700 1 -- 172.21.15.89:0/3909874801 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f41bc06b370 con 0x7f41cc102bd0 2024-03-21T13:12:29.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f41b806e4e0 msgr2=0x7f41b80709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:29.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f41b806e4e0 0x7f41b80709a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009810 tx=0x7f41c8019040 comp rx=0 tx=0).stop 2024-03-21T13:12:29.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc102bd0 msgr2=0x7f41cc1984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc102bd0 0x7f41cc1984c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009f70 tx=0x7f41bc0074a0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 shutdown_connections 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc108260 0x7f41cc19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41cc102210 0x7f41cc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f41b806e4e0 0x7f41b80709a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 --2- 172.21.15.89:0/3909874801 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc102bd0 0x7f41cc1984c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 >> 172.21.15.89:0/3909874801 conn(0x7f41cc0792e0 msgr2=0x7f41cc108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 shutdown_connections 2024-03-21T13:12:29.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:29.709+0000 7f41d39f1700 1 -- 172.21.15.89:0/3909874801 wait complete. 2024-03-21T13:12:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:29 smithi152 ceph-mon[34051]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2642598297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T13:12:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3909874801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T13:12:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:29 smithi089 ceph-mon[30095]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2642598297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T13:12:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3909874801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T13:12:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:29 smithi089 ceph-mon[39209]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2642598297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T13:12:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3909874801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T13:12:30.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149731 got 111669149735 for osd.3 2024-03-21T13:12:30.681 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:30.847 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:31.439 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953503 got 137438953506 for osd.4 2024-03-21T13:12:31.439 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:31 smithi152 ceph-mon[34051]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:31 smithi089 ceph-mon[30095]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:32.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:31 smithi089 ceph-mon[39209]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:33.060 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574896 for osd.1 2024-03-21T13:12:33.060 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:33.438 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411372 for osd.2 2024-03-21T13:12:33.438 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:33.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.625+0000 7f4ba46c4700 1 -- 172.21.15.89:0/3944059433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c1021e0 msgr2=0x7f4b9c102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.625+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/3944059433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c1021e0 0x7f4b9c102660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0098f0 tx=0x7f4b8c009c00 comp rx=0 tx=0).stop 2024-03-21T13:12:33.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- 172.21.15.89:0/3944059433 shutdown_connections 2024-03-21T13:12:33.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/3944059433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b9c102ba0 0x7f4b9c10f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/3944059433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 0x7f4b9c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/3944059433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c1021e0 0x7f4b9c102660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- 172.21.15.89:0/3944059433 >> 172.21.15.89:0/3944059433 conn(0x7f4b9c0792c0 msgr2=0x7f4b9c0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- 172.21.15.89:0/3944059433 shutdown_connections 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- 172.21.15.89:0/3944059433 wait complete. 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 Processor -- start 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- start start 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b9c1021e0 0x7f4b9c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c102ba0 0x7f4b9c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 0x7f4b9c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c111950 con 0x7f4b9c102ba0 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c1117d0 con 0x7f4b9c1021e0 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba46c4700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c111ad0 con 0x7f4b9c108290 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba1c5f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c102ba0 0x7f4b9c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba2c61700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 0x7f4b9c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba2c61700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 0x7f4b9c19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:48316/0 (socket says 172.21.15.89:48316) 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba1c5f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c102ba0 0x7f4b9c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46858/0 (socket says 172.21.15.89:46858) 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.626+0000 7f4ba2c61700 1 -- 172.21.15.89:0/1244350521 learned_addr learned my addr 172.21.15.89:0/1244350521 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba2460700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b9c1021e0 0x7f4b9c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba2c61700 1 -- 172.21.15.89:0/1244350521 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b9c1021e0 msgr2=0x7f4b9c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba2c61700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b9c1021e0 0x7f4b9c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba2c61700 1 -- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c102ba0 msgr2=0x7f4b9c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba2c61700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c102ba0 0x7f4b9c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba2c61700 1 -- 172.21.15.89:0/1244350521 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c19cf00 con 0x7f4b9c108290 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba1c5f700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c102ba0 0x7f4b9c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba2c61700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 0x7f4b9c19c7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b94007ee0 tx=0x7f4b9400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4b937fe700 1 -- 172.21.15.89:0/1244350521 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94004bb0 con 0x7f4b9c108290 2024-03-21T13:12:33.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4b937fe700 1 -- 172.21.15.89:0/1244350521 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4b94004d50 con 0x7f4b9c108290 2024-03-21T13:12:33.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c19d1f0 con 0x7f4b9c108290 2024-03-21T13:12:33.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4b937fe700 1 -- 172.21.15.89:0/1244350521 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940057a0 con 0x7f4b9c108290 2024-03-21T13:12:33.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.627+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c1a4a70 con 0x7f4b9c108290 2024-03-21T13:12:33.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.628+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c068e50 con 0x7f4b9c108290 2024-03-21T13:12:33.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.629+0000 7f4b937fe700 1 -- 172.21.15.89:0/1244350521 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f4b9401a020 con 0x7f4b9c108290 2024-03-21T13:12:33.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.629+0000 7f4b937fe700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4b8806e340 0x7f4b88070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.629+0000 7f4b937fe700 1 -- 172.21.15.89:0/1244350521 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f4b9408df60 con 0x7f4b9c108290 2024-03-21T13:12:33.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.629+0000 7f4ba2460700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4b8806e340 0x7f4b88070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.629+0000 7f4ba2460700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4b8806e340 0x7f4b88070800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009750 tx=0x7f4b98009310 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:33.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.634+0000 7f4b937fe700 1 -- 172.21.15.89:0/1244350521 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b94010070 con 0x7f4b9c108290 2024-03-21T13:12:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 -- 172.21.15.89:0/3330828754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0680102d80 msgr2=0x7f0680106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3330828754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0680102d80 0x7f0680106dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06740099c0 tx=0x7f0674009cd0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 -- 172.21.15.89:0/3330828754 shutdown_connections 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3330828754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801074b0 0x7f068010f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3330828754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06801023d0 0x7f06801027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3330828754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0680102d80 0x7f0680106dd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 -- 172.21.15.89:0/3330828754 >> 172.21.15.89:0/3330828754 conn(0x7f06800fdc40 msgr2=0x7f0680100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 -- 172.21.15.89:0/3330828754 shutdown_connections 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.799+0000 7f0685c8f700 1 -- 172.21.15.89:0/3330828754 wait complete. 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 Processor -- start 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 -- start start 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801023d0 0x7f0680074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0680102d80 0x7f06800749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06801074b0 0x7f0680074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0680111920 con 0x7f06801074b0 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06801117a0 con 0x7f0680102d80 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.800+0000 7f0685c8f700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0680111aa0 con 0x7f06801023d0 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801023d0 0x7f0680074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801023d0 0x7f0680074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:48336/0 (socket says 172.21.15.89:48336) 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 -- 172.21.15.89:0/3335930772 learned_addr learned my addr 172.21.15.89:0/3335930772 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067ffff700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06801074b0 0x7f0680074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067effd700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0680102d80 0x7f06800749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 -- 172.21.15.89:0/3335930772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0680102d80 msgr2=0x7f06800749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0680102d80 0x7f06800749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 -- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06801074b0 msgr2=0x7f0680074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06801074b0 0x7f0680074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 -- 172.21.15.89:0/3335930772 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06801a6350 con 0x7f06801023d0 2024-03-21T13:12:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067ffff700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06801074b0 0x7f0680074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.801+0000 7f067f7fe700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801023d0 0x7f0680074490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f067000c9b0 tx=0x7f067000ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.804+0000 7f067cff9700 1 -- 172.21.15.89:0/3335930772 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670007b00 con 0x7f06801023d0 2024-03-21T13:12:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.804+0000 7f067cff9700 1 -- 172.21.15.89:0/3335930772 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f067000ce40 con 0x7f06801023d0 2024-03-21T13:12:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.804+0000 7f067cff9700 1 -- 172.21.15.89:0/3335930772 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670018400 con 0x7f06801023d0 2024-03-21T13:12:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.804+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06801a6640 con 0x7f06801023d0 2024-03-21T13:12:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.804+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06801a6ba0 con 0x7f06801023d0 2024-03-21T13:12:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.805+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0680068e50 con 0x7f06801023d0 2024-03-21T13:12:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.805+0000 7f067cff9700 1 -- 172.21.15.89:0/3335930772 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f067000f450 con 0x7f06801023d0 2024-03-21T13:12:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.806+0000 7f067cff9700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f066806e340 0x7f0668070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.806+0000 7f067cff9700 1 -- 172.21.15.89:0/3335930772 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f067008e830 con 0x7f06801023d0 2024-03-21T13:12:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.806+0000 7f067effd700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f066806e340 0x7f0668070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.806+0000 7f067effd700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f066806e340 0x7f0668070800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0674000c00 tx=0x7f0674005fb0 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:33.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.810+0000 7f067cff9700 1 -- 172.21.15.89:0/3335930772 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0670014030 con 0x7f06801023d0 2024-03-21T13:12:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.839+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4b9c1a4c40 con 0x7f4b9c108290 2024-03-21T13:12:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.839+0000 7f4b937fe700 1 -- 172.21.15.89:0/1244350521 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4b9405b1d0 con 0x7f4b9c108290 2024-03-21T13:12:33.840 INFO:teuthology.orchestra.run.smithi089.stdout:171798691870 2024-03-21T13:12:33.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.841+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4b8806e340 msgr2=0x7f4b88070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.841+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4b8806e340 0x7f4b88070800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009750 tx=0x7f4b98009310 comp rx=0 tx=0).stop 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.841+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 msgr2=0x7f4b9c19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.841+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 0x7f4b9c19c7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b94007ee0 tx=0x7f4b9400d3b0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 shutdown_connections 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4b8806e340 0x7f4b88070800 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b9c108290 0x7f4b9c19c7c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b9c1021e0 0x7f4b9c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 --2- 172.21.15.89:0/1244350521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c102ba0 0x7f4b9c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 >> 172.21.15.89:0/1244350521 conn(0x7f4b9c0792c0 msgr2=0x7f4b9c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 shutdown_connections 2024-03-21T13:12:33.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:33.842+0000 7f4ba46c4700 1 -- 172.21.15.89:0/1244350521 wait complete. 2024-03-21T13:12:34.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.012+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f06801a6e30 con 0x7f06801023d0 2024-03-21T13:12:34.013 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:33 smithi089 ceph-mon[30095]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:34.013 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:33 smithi089 ceph-mon[39209]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:34.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.012+0000 7f067cff9700 1 -- 172.21.15.89:0/3335930772 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f067005bb20 con 0x7f06801023d0 2024-03-21T13:12:34.014 INFO:teuthology.orchestra.run.smithi089.stdout:34359738423 2024-03-21T13:12:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f066806e340 msgr2=0x7f0668070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f066806e340 0x7f0668070800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0674000c00 tx=0x7f0674005fb0 comp rx=0 tx=0).stop 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801023d0 msgr2=0x7f0680074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801023d0 0x7f0680074490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f067000c9b0 tx=0x7f067000ccc0 comp rx=0 tx=0).stop 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 shutdown_connections 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06801074b0 0x7f0680074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f066806e340 0x7f0668070800 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06801023d0 0x7f0680074490 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 --2- 172.21.15.89:0/3335930772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0680102d80 0x7f06800749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.014+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 >> 172.21.15.89:0/3335930772 conn(0x7f06800fdc40 msgr2=0x7f06800fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.015+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 shutdown_connections 2024-03-21T13:12:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:34.015+0000 7f0685c8f700 1 -- 172.21.15.89:0/3335930772 wait complete. 2024-03-21T13:12:34.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:33 smithi152 ceph-mon[34051]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:34.590 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738423 for osd.0 2024-03-21T13:12:34.591 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:34.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1244350521' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T13:12:34.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3335930772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T13:12:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1244350521' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T13:12:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3335930772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T13:12:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1244350521' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T13:12:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3335930772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T13:12:35.772 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691866 got 171798691870 for osd.5 2024-03-21T13:12:35.772 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:35 smithi152 ceph-mon[34051]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:35 smithi089 ceph-mon[30095]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:36.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:35 smithi089 ceph-mon[39209]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:36.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 -- 172.21.15.89:0/2883800281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c0102d80 msgr2=0x7f44c0106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 --2- 172.21.15.89:0/2883800281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c0102d80 0x7f44c0106dd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44b40099c0 tx=0x7f44b4009cd0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 -- 172.21.15.89:0/2883800281 shutdown_connections 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 --2- 172.21.15.89:0/2883800281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c01074b0 0x7f44c010f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 --2- 172.21.15.89:0/2883800281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44c01023d0 0x7f44c01027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 --2- 172.21.15.89:0/2883800281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c0102d80 0x7f44c0106dd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 -- 172.21.15.89:0/2883800281 >> 172.21.15.89:0/2883800281 conn(0x7f44c00fdc40 msgr2=0x7f44c0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 -- 172.21.15.89:0/2883800281 shutdown_connections 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.330+0000 7f44c4af7700 1 -- 172.21.15.89:0/2883800281 wait complete. 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 Processor -- start 2024-03-21T13:12:36.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 -- start start 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c01023d0 0x7f44c0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44c0102d80 0x7f44c01983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c01074b0 0x7f44c019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0111920 con 0x7f44c01074b0 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c01117a0 con 0x7f44c0102d80 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44c4af7700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0111aa0 con 0x7f44c01023d0 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c01023d0 0x7f44c0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c01023d0 0x7f44c0197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:48364/0 (socket says 172.21.15.89:48364) 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 -- 172.21.15.89:0/915450333 learned_addr learned my addr 172.21.15.89:0/915450333 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44bdd9b700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44c0102d80 0x7f44c01983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44bed9d700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c01074b0 0x7f44c019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:36.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 -- 172.21.15.89:0/915450333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44c0102d80 msgr2=0x7f44c01983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44c0102d80 0x7f44c01983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 -- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c01074b0 msgr2=0x7f44c019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c01074b0 0x7f44c019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44be59c700 1 -- 172.21.15.89:0/915450333 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c019ce90 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.331+0000 7f44bed9d700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c01074b0 0x7f44c019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.332+0000 7f44be59c700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c01023d0 0x7f44c0197e80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f44b000d980 tx=0x7f44b000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.332+0000 7f44af7fe700 1 -- 172.21.15.89:0/915450333 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0017070 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.332+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c019d180 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.332+0000 7f44af7fe700 1 -- 172.21.15.89:0/915450333 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f44b0005020 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.332+0000 7f44af7fe700 1 -- 172.21.15.89:0/915450333 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0009a30 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.332+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c01a4a00 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.333+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c0068e50 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.333+0000 7f44af7fe700 1 -- 172.21.15.89:0/915450333 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f44b0009bd0 con 0x7f44c01023d0 2024-03-21T13:12:36.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.334+0000 7f44af7fe700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f44a806e340 0x7f44a8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.334+0000 7f44af7fe700 1 -- 172.21.15.89:0/915450333 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f44b008e820 con 0x7f44c01023d0 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.334+0000 7f44bdd9b700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f44a806e340 0x7f44a8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.334+0000 7f44bdd9b700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f44a806e340 0x7f44a8070800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44b4000c00 tx=0x7f44b4005fb0 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.338+0000 7f44af7fe700 1 -- 172.21.15.89:0/915450333 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44b0020020 con 0x7f44c01023d0 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.539+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f44c01a4bd0 con 0x7f44c01023d0 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.539+0000 7f44af7fe700 1 -- 172.21.15.89:0/915450333 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f44b005ba90 con 0x7f44c01023d0 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f44a806e340 msgr2=0x7f44a8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f44a806e340 0x7f44a8070800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44b4000c00 tx=0x7f44b4005fb0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c01023d0 msgr2=0x7f44c0197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c01023d0 0x7f44c0197e80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f44b000d980 tx=0x7f44b000dc90 comp rx=0 tx=0).stop 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 shutdown_connections 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c01074b0 0x7f44c019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f44a806e340 0x7f44a8070800 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44c01023d0 0x7f44c0197e80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 --2- 172.21.15.89:0/915450333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44c0102d80 0x7f44c01983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:36.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 >> 172.21.15.89:0/915450333 conn(0x7f44c00fdc40 msgr2=0x7f44c00fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:36.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 shutdown_connections 2024-03-21T13:12:36.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:36.541+0000 7f44c4af7700 1 -- 172.21.15.89:0/915450333 wait complete. 2024-03-21T13:12:36.961 INFO:teuthology.orchestra.run.smithi089.stdout:206158430233 2024-03-21T13:12:37.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/915450333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T13:12:37.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/915450333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T13:12:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/915450333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T13:12:37.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430229 got 206158430233 for osd.6 2024-03-21T13:12:37.587 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:37.795 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:12:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:37.796 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:12:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[76445]: ts=2024-03-21T13:12:37.745Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T13:12:37.796 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:12:37 smithi089 bash[76203]: 547cc80e49c368a0fb207d37a54965383fff0e038e9f30c75a37c1487c60daf4 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 -- 172.21.15.89:0/264379587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 msgr2=0x7f92101027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 --2- 172.21.15.89:0/264379587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f92101027b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f920c0099c0 tx=0x7f920c009cd0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 -- 172.21.15.89:0/264379587 shutdown_connections 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 --2- 172.21.15.89:0/264379587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92101074b0 0x7f921010f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 --2- 172.21.15.89:0/264379587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f92101027b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 --2- 172.21.15.89:0/264379587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9210102d80 0x7f9210106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 -- 172.21.15.89:0/264379587 >> 172.21.15.89:0/264379587 conn(0x7f92100fdc40 msgr2=0x7f9210100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 -- 172.21.15.89:0/264379587 shutdown_connections 2024-03-21T13:12:38.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 -- 172.21.15.89:0/264379587 wait complete. 2024-03-21T13:12:38.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 Processor -- start 2024-03-21T13:12:38.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.042+0000 7f921789b700 1 -- start start 2024-03-21T13:12:38.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f921789b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f9210197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:38.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f921789b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9210102d80 0x7f92101983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:38.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f921789b700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92101074b0 0x7f921019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f921789b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9210111900 con 0x7f92101023d0 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f921789b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9210111780 con 0x7f9210102d80 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f921789b700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9210111a80 con 0x7f92101074b0 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f9210197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f9210197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36566/0 (socket says 172.21.15.89:36566) 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 -- 172.21.15.89:0/989714222 learned_addr learned my addr 172.21.15.89:0/989714222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215e38700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92101074b0 0x7f921019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9214e36700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9210102d80 0x7f92101983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 -- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92101074b0 msgr2=0x7f921019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92101074b0 0x7f921019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 -- 172.21.15.89:0/989714222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9210102d80 msgr2=0x7f92101983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9210102d80 0x7f92101983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 -- 172.21.15.89:0/989714222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921019ce90 con 0x7f92101023d0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.043+0000 7f9215637700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f9210197e80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f920c006010 tx=0x7f920c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.044+0000 7f92067fc700 1 -- 172.21.15.89:0/989714222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c026070 con 0x7f92101023d0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.044+0000 7f92067fc700 1 -- 172.21.15.89:0/989714222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f920c003e50 con 0x7f92101023d0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.044+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921019d120 con 0x7f92101023d0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.044+0000 7f92067fc700 1 -- 172.21.15.89:0/989714222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c00fb30 con 0x7f92101023d0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.044+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92101a4a00 con 0x7f92101023d0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.045+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92101038b0 con 0x7f92101023d0 2024-03-21T13:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.045+0000 7f92067fc700 1 -- 172.21.15.89:0/989714222 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f920c029070 con 0x7f92101023d0 2024-03-21T13:12:38.048 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:12:37 smithi089 systemd[1]: Started Ceph alertmanager.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:12:38.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:37 smithi089 ceph-mon[30095]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:38.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:37 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:38.049 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:37 smithi089 ceph-mon[39209]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:38.049 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:37 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.046+0000 7f92067fc700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f91fc06e340 0x7f91fc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.046+0000 7f92067fc700 1 -- 172.21.15.89:0/989714222 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f920c08f230 con 0x7f92101023d0 2024-03-21T13:12:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.046+0000 7f9214e36700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f91fc06e340 0x7f91fc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.046+0000 7f9214e36700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f91fc06e340 0x7f91fc070800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92100ffe50 tx=0x7f9200009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:38.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.051+0000 7f92067fc700 1 -- 172.21.15.89:0/989714222 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f920c013070 con 0x7f92101023d0 2024-03-21T13:12:38.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:37 smithi152 ceph-mon[34051]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:38.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:37 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:38.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:12:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:12:38.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.255+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f92101991e0 con 0x7f92101023d0 2024-03-21T13:12:38.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.256+0000 7f92067fc700 1 -- 172.21.15.89:0/989714222 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f920c05c4a0 con 0x7f92101023d0 2024-03-21T13:12:38.257 INFO:teuthology.orchestra.run.smithi089.stdout:240518168597 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f91fc06e340 msgr2=0x7f91fc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f91fc06e340 0x7f91fc070800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92100ffe50 tx=0x7f9200009260 comp rx=0 tx=0).stop 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 msgr2=0x7f9210197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f9210197e80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f920c006010 tx=0x7f920c0045e0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 shutdown_connections 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f91fc06e340 0x7f91fc070800 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92101074b0 0x7f921019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92101023d0 0x7f9210197e80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 --2- 172.21.15.89:0/989714222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9210102d80 0x7f92101983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:38.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 >> 172.21.15.89:0/989714222 conn(0x7f92100fdc40 msgr2=0x7f92100fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:38.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 shutdown_connections 2024-03-21T13:12:38.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:38.258+0000 7f921789b700 1 -- 172.21.15.89:0/989714222 wait complete. 2024-03-21T13:12:38.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:12:38] ENGINE Bus STOPPING 2024-03-21T13:12:38.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:12:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:12:38.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:12:38] ENGINE Bus STOPPED 2024-03-21T13:12:38.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:12:38] ENGINE Bus STARTING 2024-03-21T13:12:38.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:12:38] ENGINE Serving on http://:::9283 2024-03-21T13:12:38.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:12:38] ENGINE Bus STARTED 2024-03-21T13:12:38.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168592 got 240518168597 for osd.7 2024-03-21T13:12:38.972 DEBUG:teuthology.parallel:result is None 2024-03-21T13:12:38.973 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T13:12:38.973 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:12:39.217 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T13:12:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:12:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:12:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: Deploying daemon grafana.a on smithi152 2024-03-21T13:12:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/989714222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:12:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: Deploying daemon grafana.a on smithi152 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/989714222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:12:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T13:12:39.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:39.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: Deploying daemon grafana.a on smithi152 2024-03-21T13:12:39.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/989714222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T13:12:40.035 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:12:39 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[76445]: ts=2024-03-21T13:12:39.718Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000013327s 2024-03-21T13:12:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:40 smithi089 ceph-mon[30095]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:40 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:40 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:40 smithi089 ceph-mon[39209]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:40 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:40 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:40 smithi152 ceph-mon[34051]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:40 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:40 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 -- 172.21.15.89:0/3026896730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd70102ba0 msgr2=0x7fdd7010efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 --2- 172.21.15.89:0/3026896730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd70102ba0 0x7fdd7010efe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800b0a0 tx=0x7fdd6800b3b0 comp rx=0 tx=0).stop 2024-03-21T13:12:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 -- 172.21.15.89:0/3026896730 shutdown_connections 2024-03-21T13:12:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 --2- 172.21.15.89:0/3026896730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd70102ba0 0x7fdd7010efe0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 --2- 172.21.15.89:0/3026896730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd701081c0 0x7fdd701085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 --2- 172.21.15.89:0/3026896730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd701021e0 0x7fdd70102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 -- 172.21.15.89:0/3026896730 >> 172.21.15.89:0/3026896730 conn(0x7fdd700792c0 msgr2=0x7fdd700796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 -- 172.21.15.89:0/3026896730 shutdown_connections 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 -- 172.21.15.89:0/3026896730 wait complete. 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 Processor -- start 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.827+0000 7fdd7857d700 1 -- start start 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd701021e0 0x7fdd70074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd70102ba0 0x7fdd700749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd701081c0 0x7fdd70074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd70111910 con 0x7fdd701081c0 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd70111790 con 0x7fdd70102ba0 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd70111a90 con 0x7fdd701021e0 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd701021e0 0x7fdd70074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd701021e0 0x7fdd70074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:45810/0 (socket says 172.21.15.89:45810) 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 -- 172.21.15.89:0/835608583 learned_addr learned my addr 172.21.15.89:0/835608583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76b1a700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd701081c0 0x7fdd70074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd75b18700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd70102ba0 0x7fdd700749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 -- 172.21.15.89:0/835608583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd70102ba0 msgr2=0x7fdd700749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd70102ba0 0x7fdd700749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 -- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd701081c0 msgr2=0x7fdd70074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd701081c0 0x7fdd70074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 -- 172.21.15.89:0/835608583 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd701a6280 con 0x7fdd701021e0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd76319700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd701021e0 0x7fdd70074490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000eb30 tx=0x7fdd6000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd677fe700 1 -- 172.21.15.89:0/835608583 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000cbd0 con 0x7fdd701021e0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd701a6570 con 0x7fdd701021e0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd677fe700 1 -- 172.21.15.89:0/835608583 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdd600044a0 con 0x7fdd701021e0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd677fe700 1 -- 172.21.15.89:0/835608583 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60003d40 con 0x7fdd701021e0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.828+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd701a6b00 con 0x7fdd701021e0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.829+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd70068e50 con 0x7fdd701021e0 2024-03-21T13:12:40.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.830+0000 7fdd677fe700 1 -- 172.21.15.89:0/835608583 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fdd60003ee0 con 0x7fdd701021e0 2024-03-21T13:12:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.830+0000 7fdd677fe700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fdd5c06e340 0x7fdd5c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.830+0000 7fdd75b18700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fdd5c06e340 0x7fdd5c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.830+0000 7fdd677fe700 1 -- 172.21.15.89:0/835608583 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fdd6008eb00 con 0x7fdd701021e0 2024-03-21T13:12:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.831+0000 7fdd75b18700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fdd5c06e340 0x7fdd5c070800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdd70112390 tx=0x7fdd6c00b3c0 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:40.835+0000 7fdd677fe700 1 -- 172.21.15.89:0/835608583 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd6005bd70 con 0x7fdd701021e0 2024-03-21T13:12:41.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.026+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 --> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd70107490 con 0x7fdd5c06e340 2024-03-21T13:12:41.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.036+0000 7fdd677fe700 1 -- 172.21.15.89:0/835608583 <== mgr.24458 v2:172.21.15.89:6800/2642177467 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346467 (secure 0 0 0) 0x7fdd70107490 con 0x7fdd5c06e340 2024-03-21T13:12:41.042 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.039+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fdd5c06e340 msgr2=0x7fdd5c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.039+0000 7fdd7857d700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fdd5c06e340 0x7fdd5c070800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdd70112390 tx=0x7fdd6c00b3c0 comp rx=0 tx=0).stop 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.039+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd701021e0 msgr2=0x7fdd70074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.039+0000 7fdd7857d700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd701021e0 0x7fdd70074490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000eb30 tx=0x7fdd6000ee40 comp rx=0 tx=0).stop 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 shutdown_connections 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd701081c0 0x7fdd70074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fdd5c06e340 0x7fdd5c070800 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd701021e0 0x7fdd70074490 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 --2- 172.21.15.89:0/835608583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd70102ba0 0x7fdd700749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 >> 172.21.15.89:0/835608583 conn(0x7fdd700792c0 msgr2=0x7fdd70105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 shutdown_connections 2024-03-21T13:12:41.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:41.040+0000 7fdd7857d700 1 -- 172.21.15.89:0/835608583 wait complete. 2024-03-21T13:12:41.047 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-03-21T13:12:41.649 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-21T13:12:40.763808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1013,"num_read_kb":859,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":257556,"kb_used_data":8124,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":749703660,"statfs":{"total":767960285184,"available":767696547840,"internally_reserved":0,"allocated":8318976,"data_stored":4488304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002465"},"pg_stats":[{"pgid":"6.1a","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749249+0000","last_change":"2024-03-21T13:11:22.327003+0000","last_active":"2024-03-21T13:11:54.749249+0000","last_peered":"2024-03-21T13:11:54.749249+0000","last_clean":"2024-03-21T13:11:54.749249+0000","last_became_active":"2024-03-21T13:11:22.325840+0000","last_became_peered":"2024-03-21T13:11:22.325840+0000","last_unstale":"2024-03-21T13:11:54.749249+0000","last_undegraded":"2024-03-21T13:11:54.749249+0000","last_fullsized":"2024-03-21T13:11:54.749249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:26:27.565490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997987+0000","last_change":"2024-03-21T13:11:20.315075+0000","last_active":"2024-03-21T13:11:54.997987+0000","last_peered":"2024-03-21T13:11:54.997987+0000","last_clean":"2024-03-21T13:11:54.997987+0000","last_became_active":"2024-03-21T13:11:20.314208+0000","last_became_peered":"2024-03-21T13:11:20.314208+0000","last_unstale":"2024-03-21T13:11:54.997987+0000","last_undegraded":"2024-03-21T13:11:54.997987+0000","last_fullsized":"2024-03-21T13:11:54.997987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:40:57.353281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998664+0000","last_change":"2024-03-21T13:11:16.144659+0000","last_active":"2024-03-21T13:11:54.998664+0000","last_peered":"2024-03-21T13:11:54.998664+0000","last_clean":"2024-03-21T13:11:54.998664+0000","last_became_active":"2024-03-21T13:11:16.144369+0000","last_became_peered":"2024-03-21T13:11:16.144369+0000","last_unstale":"2024-03-21T13:11:54.998664+0000","last_undegraded":"2024-03-21T13:11:54.998664+0000","last_fullsized":"2024-03-21T13:11:54.998664+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:16:46.294733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748084+0000","last_change":"2024-03-21T13:11:18.294733+0000","last_active":"2024-03-21T13:11:54.748084+0000","last_peered":"2024-03-21T13:11:54.748084+0000","last_clean":"2024-03-21T13:11:54.748084+0000","last_became_active":"2024-03-21T13:11:18.294374+0000","last_became_peered":"2024-03-21T13:11:18.294374+0000","last_unstale":"2024-03-21T13:11:54.748084+0000","last_undegraded":"2024-03-21T13:11:54.748084+0000","last_fullsized":"2024-03-21T13:11:54.748084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:45:22.942768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749106+0000","last_change":"2024-03-21T13:11:22.518891+0000","last_active":"2024-03-21T13:11:54.749106+0000","last_peered":"2024-03-21T13:11:54.749106+0000","last_clean":"2024-03-21T13:11:54.749106+0000","last_became_active":"2024-03-21T13:11:22.518517+0000","last_became_peered":"2024-03-21T13:11:22.518517+0000","last_unstale":"2024-03-21T13:11:54.749106+0000","last_undegraded":"2024-03-21T13:11:54.749106+0000","last_fullsized":"2024-03-21T13:11:54.749106+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:13:41.048630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748680+0000","last_change":"2024-03-21T13:11:20.312628+0000","last_active":"2024-03-21T13:11:54.748680+0000","last_peered":"2024-03-21T13:11:54.748680+0000","last_clean":"2024-03-21T13:11:54.748680+0000","last_became_active":"2024-03-21T13:11:20.312336+0000","last_became_peered":"2024-03-21T13:11:20.312336+0000","last_unstale":"2024-03-21T13:11:54.748680+0000","last_undegraded":"2024-03-21T13:11:54.748680+0000","last_fullsized":"2024-03-21T13:11:54.748680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:27:15.221560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748402+0000","last_change":"2024-03-21T13:11:16.146703+0000","last_active":"2024-03-21T13:11:54.748402+0000","last_peered":"2024-03-21T13:11:54.748402+0000","last_clean":"2024-03-21T13:11:54.748402+0000","last_became_active":"2024-03-21T13:11:16.146389+0000","last_became_peered":"2024-03-21T13:11:16.146389+0000","last_unstale":"2024-03-21T13:11:54.748402+0000","last_undegraded":"2024-03-21T13:11:54.748402+0000","last_fullsized":"2024-03-21T13:11:54.748402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:44:59.580339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748790+0000","last_change":"2024-03-21T13:11:18.303525+0000","last_active":"2024-03-21T13:11:54.748790+0000","last_peered":"2024-03-21T13:11:54.748790+0000","last_clean":"2024-03-21T13:11:54.748790+0000","last_became_active":"2024-03-21T13:11:18.303103+0000","last_became_peered":"2024-03-21T13:11:18.303103+0000","last_unstale":"2024-03-21T13:11:54.748790+0000","last_undegraded":"2024-03-21T13:11:54.748790+0000","last_fullsized":"2024-03-21T13:11:54.748790+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:18:41.779791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997858+0000","last_change":"2024-03-21T13:11:22.319817+0000","last_active":"2024-03-21T13:11:54.997858+0000","last_peered":"2024-03-21T13:11:54.997858+0000","last_clean":"2024-03-21T13:11:54.997858+0000","last_became_active":"2024-03-21T13:11:22.319493+0000","last_became_peered":"2024-03-21T13:11:22.319493+0000","last_unstale":"2024-03-21T13:11:54.997858+0000","last_undegraded":"2024-03-21T13:11:54.997858+0000","last_fullsized":"2024-03-21T13:11:54.997858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:04:24.024296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846413+0000","last_change":"2024-03-21T13:11:20.321173+0000","last_active":"2024-03-21T13:11:54.846413+0000","last_peered":"2024-03-21T13:11:54.846413+0000","last_clean":"2024-03-21T13:11:54.846413+0000","last_became_active":"2024-03-21T13:11:20.320841+0000","last_became_peered":"2024-03-21T13:11:20.320841+0000","last_unstale":"2024-03-21T13:11:54.846413+0000","last_undegraded":"2024-03-21T13:11:54.846413+0000","last_fullsized":"2024-03-21T13:11:54.846413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:29:23.173402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846746+0000","last_change":"2024-03-21T13:11:16.145158+0000","last_active":"2024-03-21T13:11:54.846746+0000","last_peered":"2024-03-21T13:11:54.846746+0000","last_clean":"2024-03-21T13:11:54.846746+0000","last_became_active":"2024-03-21T13:11:16.144851+0000","last_became_peered":"2024-03-21T13:11:16.144851+0000","last_unstale":"2024-03-21T13:11:54.846746+0000","last_undegraded":"2024-03-21T13:11:54.846746+0000","last_fullsized":"2024-03-21T13:11:54.846746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:22:50.803508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749497+0000","last_change":"2024-03-21T13:11:18.300903+0000","last_active":"2024-03-21T13:11:54.749497+0000","last_peered":"2024-03-21T13:11:54.749497+0000","last_clean":"2024-03-21T13:11:54.749497+0000","last_became_active":"2024-03-21T13:11:18.300472+0000","last_became_peered":"2024-03-21T13:11:18.300472+0000","last_unstale":"2024-03-21T13:11:54.749497+0000","last_undegraded":"2024-03-21T13:11:54.749497+0000","last_fullsized":"2024-03-21T13:11:54.749497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:19:02.076821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846248+0000","last_change":"2024-03-21T13:11:22.327344+0000","last_active":"2024-03-21T13:11:54.846248+0000","last_peered":"2024-03-21T13:11:54.846248+0000","last_clean":"2024-03-21T13:11:54.846248+0000","last_became_active":"2024-03-21T13:11:22.327000+0000","last_became_peered":"2024-03-21T13:11:22.327000+0000","last_unstale":"2024-03-21T13:11:54.846248+0000","last_undegraded":"2024-03-21T13:11:54.846248+0000","last_fullsized":"2024-03-21T13:11:54.846248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:18:33.539629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856244+0000","last_change":"2024-03-21T13:11:20.316689+0000","last_active":"2024-03-21T13:11:54.856244+0000","last_peered":"2024-03-21T13:11:54.856244+0000","last_clean":"2024-03-21T13:11:54.856244+0000","last_became_active":"2024-03-21T13:11:20.316301+0000","last_became_peered":"2024-03-21T13:11:20.316301+0000","last_unstale":"2024-03-21T13:11:54.856244+0000","last_undegraded":"2024-03-21T13:11:54.856244+0000","last_fullsized":"2024-03-21T13:11:54.856244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:44:08.398213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846704+0000","last_change":"2024-03-21T13:11:16.146296+0000","last_active":"2024-03-21T13:11:54.846704+0000","last_peered":"2024-03-21T13:11:54.846704+0000","last_clean":"2024-03-21T13:11:54.846704+0000","last_became_active":"2024-03-21T13:11:16.145782+0000","last_became_peered":"2024-03-21T13:11:16.145782+0000","last_unstale":"2024-03-21T13:11:54.846704+0000","last_undegraded":"2024-03-21T13:11:54.846704+0000","last_fullsized":"2024-03-21T13:11:54.846704+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:26:01.867415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":51,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748455+0000","last_change":"2024-03-21T13:11:44.604290+0000","last_active":"2024-03-21T13:11:54.748455+0000","last_peered":"2024-03-21T13:11:54.748455+0000","last_clean":"2024-03-21T13:11:54.748455+0000","last_became_active":"2024-03-21T13:11:44.593947+0000","last_became_peered":"2024-03-21T13:11:44.593947+0000","last_unstale":"2024-03-21T13:11:54.748455+0000","last_undegraded":"2024-03-21T13:11:54.748455+0000","last_fullsized":"2024-03-21T13:11:54.748455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:35.758885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749237+0000","last_change":"2024-03-21T13:11:22.517477+0000","last_active":"2024-03-21T13:11:54.749237+0000","last_peered":"2024-03-21T13:11:54.749237+0000","last_clean":"2024-03-21T13:11:54.749237+0000","last_became_active":"2024-03-21T13:11:22.517162+0000","last_became_peered":"2024-03-21T13:11:22.517162+0000","last_unstale":"2024-03-21T13:11:54.749237+0000","last_undegraded":"2024-03-21T13:11:54.749237+0000","last_fullsized":"2024-03-21T13:11:54.749237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:21:23.488718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999001+0000","last_change":"2024-03-21T13:11:20.311348+0000","last_active":"2024-03-21T13:11:54.999001+0000","last_peered":"2024-03-21T13:11:54.999001+0000","last_clean":"2024-03-21T13:11:54.999001+0000","last_became_active":"2024-03-21T13:11:20.310699+0000","last_became_peered":"2024-03-21T13:11:20.310699+0000","last_unstale":"2024-03-21T13:11:54.999001+0000","last_undegraded":"2024-03-21T13:11:54.999001+0000","last_fullsized":"2024-03-21T13:11:54.999001+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:28:35.331844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998326+0000","last_change":"2024-03-21T13:11:16.142611+0000","last_active":"2024-03-21T13:11:54.998326+0000","last_peered":"2024-03-21T13:11:54.998326+0000","last_clean":"2024-03-21T13:11:54.998326+0000","last_became_active":"2024-03-21T13:11:16.142299+0000","last_became_peered":"2024-03-21T13:11:16.142299+0000","last_unstale":"2024-03-21T13:11:54.998326+0000","last_undegraded":"2024-03-21T13:11:54.998326+0000","last_fullsized":"2024-03-21T13:11:54.998326+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:52:24.309637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744077+0000","last_change":"2024-03-21T13:11:18.302257+0000","last_active":"2024-03-21T13:11:54.744077+0000","last_peered":"2024-03-21T13:11:54.744077+0000","last_clean":"2024-03-21T13:11:54.744077+0000","last_became_active":"2024-03-21T13:11:18.301913+0000","last_became_peered":"2024-03-21T13:11:18.301913+0000","last_unstale":"2024-03-21T13:11:54.744077+0000","last_undegraded":"2024-03-21T13:11:54.744077+0000","last_fullsized":"2024-03-21T13:11:54.744077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:59:05.674276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748903+0000","last_change":"2024-03-21T13:11:22.517065+0000","last_active":"2024-03-21T13:11:54.748903+0000","last_peered":"2024-03-21T13:11:54.748903+0000","last_clean":"2024-03-21T13:11:54.748903+0000","last_became_active":"2024-03-21T13:11:22.516450+0000","last_became_peered":"2024-03-21T13:11:22.516450+0000","last_unstale":"2024-03-21T13:11:54.748903+0000","last_undegraded":"2024-03-21T13:11:54.748903+0000","last_fullsized":"2024-03-21T13:11:54.748903+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:51:12.591461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748305+0000","last_change":"2024-03-21T13:11:20.314859+0000","last_active":"2024-03-21T13:11:54.748305+0000","last_peered":"2024-03-21T13:11:54.748305+0000","last_clean":"2024-03-21T13:11:54.748305+0000","last_became_active":"2024-03-21T13:11:20.314529+0000","last_became_peered":"2024-03-21T13:11:20.314529+0000","last_unstale":"2024-03-21T13:11:54.748305+0000","last_undegraded":"2024-03-21T13:11:54.748305+0000","last_fullsized":"2024-03-21T13:11:54.748305+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:19:20.260436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749612+0000","last_change":"2024-03-21T13:11:16.145821+0000","last_active":"2024-03-21T13:11:54.749612+0000","last_peered":"2024-03-21T13:11:54.749612+0000","last_clean":"2024-03-21T13:11:54.749612+0000","last_became_active":"2024-03-21T13:11:16.145518+0000","last_became_peered":"2024-03-21T13:11:16.145518+0000","last_unstale":"2024-03-21T13:11:54.749612+0000","last_undegraded":"2024-03-21T13:11:54.749612+0000","last_fullsized":"2024-03-21T13:11:54.749612+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:28:34.069607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":52,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.747627+0000","last_change":"2024-03-21T13:11:18.301984+0000","last_active":"2024-03-21T13:11:54.747627+0000","last_peered":"2024-03-21T13:11:54.747627+0000","last_clean":"2024-03-21T13:11:54.747627+0000","last_became_active":"2024-03-21T13:11:18.301672+0000","last_became_peered":"2024-03-21T13:11:18.301672+0000","last_unstale":"2024-03-21T13:11:54.747627+0000","last_undegraded":"2024-03-21T13:11:54.747627+0000","last_fullsized":"2024-03-21T13:11:54.747627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:46:01.292003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856487+0000","last_change":"2024-03-21T13:11:22.325445+0000","last_active":"2024-03-21T13:11:54.856487+0000","last_peered":"2024-03-21T13:11:54.856487+0000","last_clean":"2024-03-21T13:11:54.856487+0000","last_became_active":"2024-03-21T13:11:22.324838+0000","last_became_peered":"2024-03-21T13:11:22.324838+0000","last_unstale":"2024-03-21T13:11:54.856487+0000","last_undegraded":"2024-03-21T13:11:54.856487+0000","last_fullsized":"2024-03-21T13:11:54.856487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:09:10.134796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210842+0000","last_change":"2024-03-21T13:11:20.312952+0000","last_active":"2024-03-21T13:12:31.210842+0000","last_peered":"2024-03-21T13:12:31.210842+0000","last_clean":"2024-03-21T13:12:31.210842+0000","last_became_active":"2024-03-21T13:11:20.312083+0000","last_became_peered":"2024-03-21T13:11:20.312083+0000","last_unstale":"2024-03-21T13:12:31.210842+0000","last_undegraded":"2024-03-21T13:12:31.210842+0000","last_fullsized":"2024-03-21T13:12:31.210842+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:55:31.135729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998065+0000","last_change":"2024-03-21T13:11:16.150340+0000","last_active":"2024-03-21T13:11:54.998065+0000","last_peered":"2024-03-21T13:11:54.998065+0000","last_clean":"2024-03-21T13:11:54.998065+0000","last_became_active":"2024-03-21T13:11:16.150025+0000","last_became_peered":"2024-03-21T13:11:16.150025+0000","last_unstale":"2024-03-21T13:11:54.998065+0000","last_undegraded":"2024-03-21T13:11:54.998065+0000","last_fullsized":"2024-03-21T13:11:54.998065+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:53:34.844020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998194+0000","last_change":"2024-03-21T13:11:18.517918+0000","last_active":"2024-03-21T13:11:54.998194+0000","last_peered":"2024-03-21T13:11:54.998194+0000","last_clean":"2024-03-21T13:11:54.998194+0000","last_became_active":"2024-03-21T13:11:18.517378+0000","last_became_peered":"2024-03-21T13:11:18.517378+0000","last_unstale":"2024-03-21T13:11:54.998194+0000","last_undegraded":"2024-03-21T13:11:54.998194+0000","last_fullsized":"2024-03-21T13:11:54.998194+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:46:00.357781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999112+0000","last_change":"2024-03-21T13:11:22.319805+0000","last_active":"2024-03-21T13:11:54.999112+0000","last_peered":"2024-03-21T13:11:54.999112+0000","last_clean":"2024-03-21T13:11:54.999112+0000","last_became_active":"2024-03-21T13:11:22.319425+0000","last_became_peered":"2024-03-21T13:11:22.319425+0000","last_unstale":"2024-03-21T13:11:54.999112+0000","last_undegraded":"2024-03-21T13:11:54.999112+0000","last_fullsized":"2024-03-21T13:11:54.999112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:27:33.427424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998758+0000","last_change":"2024-03-21T13:11:20.315555+0000","last_active":"2024-03-21T13:11:54.998758+0000","last_peered":"2024-03-21T13:11:54.998758+0000","last_clean":"2024-03-21T13:11:54.998758+0000","last_became_active":"2024-03-21T13:11:20.315111+0000","last_became_peered":"2024-03-21T13:11:20.315111+0000","last_unstale":"2024-03-21T13:11:54.998758+0000","last_undegraded":"2024-03-21T13:11:54.998758+0000","last_fullsized":"2024-03-21T13:11:54.998758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:55:32.803240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746041+0000","last_change":"2024-03-21T13:11:16.152332+0000","last_active":"2024-03-21T13:11:54.746041+0000","last_peered":"2024-03-21T13:11:54.746041+0000","last_clean":"2024-03-21T13:11:54.746041+0000","last_became_active":"2024-03-21T13:11:16.152009+0000","last_became_peered":"2024-03-21T13:11:16.152009+0000","last_unstale":"2024-03-21T13:11:54.746041+0000","last_undegraded":"2024-03-21T13:11:54.746041+0000","last_fullsized":"2024-03-21T13:11:54.746041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:36:46.651590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998340+0000","last_change":"2024-03-21T13:11:18.300099+0000","last_active":"2024-03-21T13:11:54.998340+0000","last_peered":"2024-03-21T13:11:54.998340+0000","last_clean":"2024-03-21T13:11:54.998340+0000","last_became_active":"2024-03-21T13:11:18.299744+0000","last_became_peered":"2024-03-21T13:11:18.299744+0000","last_unstale":"2024-03-21T13:11:54.998340+0000","last_undegraded":"2024-03-21T13:11:54.998340+0000","last_fullsized":"2024-03-21T13:11:54.998340+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:56:38.686906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856238+0000","last_change":"2024-03-21T13:11:22.323899+0000","last_active":"2024-03-21T13:11:54.856238+0000","last_peered":"2024-03-21T13:11:54.856238+0000","last_clean":"2024-03-21T13:11:54.856238+0000","last_became_active":"2024-03-21T13:11:22.323415+0000","last_became_peered":"2024-03-21T13:11:22.323415+0000","last_unstale":"2024-03-21T13:11:54.856238+0000","last_undegraded":"2024-03-21T13:11:54.856238+0000","last_fullsized":"2024-03-21T13:11:54.856238+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:27:57.543398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997604+0000","last_change":"2024-03-21T13:11:20.318426+0000","last_active":"2024-03-21T13:11:54.997604+0000","last_peered":"2024-03-21T13:11:54.997604+0000","last_clean":"2024-03-21T13:11:54.997604+0000","last_became_active":"2024-03-21T13:11:20.318028+0000","last_became_peered":"2024-03-21T13:11:20.318028+0000","last_unstale":"2024-03-21T13:11:54.997604+0000","last_undegraded":"2024-03-21T13:11:54.997604+0000","last_fullsized":"2024-03-21T13:11:54.997604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:37:41.487058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998764+0000","last_change":"2024-03-21T13:11:16.154151+0000","last_active":"2024-03-21T13:11:54.998764+0000","last_peered":"2024-03-21T13:11:54.998764+0000","last_clean":"2024-03-21T13:11:54.998764+0000","last_became_active":"2024-03-21T13:11:16.153801+0000","last_became_peered":"2024-03-21T13:11:16.153801+0000","last_unstale":"2024-03-21T13:11:54.998764+0000","last_undegraded":"2024-03-21T13:11:54.998764+0000","last_fullsized":"2024-03-21T13:11:54.998764+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:31:39.087490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":35,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746148+0000","last_change":"2024-03-21T13:11:18.300432+0000","last_active":"2024-03-21T13:11:54.746148+0000","last_peered":"2024-03-21T13:11:54.746148+0000","last_clean":"2024-03-21T13:11:54.746148+0000","last_became_active":"2024-03-21T13:11:18.300027+0000","last_became_peered":"2024-03-21T13:11:18.300027+0000","last_unstale":"2024-03-21T13:11:54.746148+0000","last_undegraded":"2024-03-21T13:11:54.746148+0000","last_fullsized":"2024-03-21T13:11:54.746148+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:21:09.035014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749009+0000","last_change":"2024-03-21T13:11:22.517547+0000","last_active":"2024-03-21T13:11:54.749009+0000","last_peered":"2024-03-21T13:11:54.749009+0000","last_clean":"2024-03-21T13:11:54.749009+0000","last_became_active":"2024-03-21T13:11:22.517064+0000","last_became_peered":"2024-03-21T13:11:22.517064+0000","last_unstale":"2024-03-21T13:11:54.749009+0000","last_undegraded":"2024-03-21T13:11:54.749009+0000","last_fullsized":"2024-03-21T13:11:54.749009+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:11:23.292658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856450+0000","last_change":"2024-03-21T13:11:20.309847+0000","last_active":"2024-03-21T13:11:54.856450+0000","last_peered":"2024-03-21T13:11:54.856450+0000","last_clean":"2024-03-21T13:11:54.856450+0000","last_became_active":"2024-03-21T13:11:20.309263+0000","last_became_peered":"2024-03-21T13:11:20.309263+0000","last_unstale":"2024-03-21T13:11:54.856450+0000","last_undegraded":"2024-03-21T13:11:54.856450+0000","last_fullsized":"2024-03-21T13:11:54.856450+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:25.930000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846651+0000","last_change":"2024-03-21T13:11:16.145714+0000","last_active":"2024-03-21T13:11:54.846651+0000","last_peered":"2024-03-21T13:11:54.846651+0000","last_clean":"2024-03-21T13:11:54.846651+0000","last_became_active":"2024-03-21T13:11:16.145431+0000","last_became_peered":"2024-03-21T13:11:16.145431+0000","last_unstale":"2024-03-21T13:11:54.846651+0000","last_undegraded":"2024-03-21T13:11:54.846651+0000","last_fullsized":"2024-03-21T13:11:54.846651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:16:26.774169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748665+0000","last_change":"2024-03-21T13:11:18.515931+0000","last_active":"2024-03-21T13:11:54.748665+0000","last_peered":"2024-03-21T13:11:54.748665+0000","last_clean":"2024-03-21T13:11:54.748665+0000","last_became_active":"2024-03-21T13:11:18.515623+0000","last_became_peered":"2024-03-21T13:11:18.515623+0000","last_unstale":"2024-03-21T13:11:54.748665+0000","last_undegraded":"2024-03-21T13:11:54.748665+0000","last_fullsized":"2024-03-21T13:11:54.748665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:12:32.702235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997497+0000","last_change":"2024-03-21T13:11:22.317379+0000","last_active":"2024-03-21T13:11:54.997497+0000","last_peered":"2024-03-21T13:11:54.997497+0000","last_clean":"2024-03-21T13:11:54.997497+0000","last_became_active":"2024-03-21T13:11:22.316810+0000","last_became_peered":"2024-03-21T13:11:22.316810+0000","last_unstale":"2024-03-21T13:11:54.997497+0000","last_undegraded":"2024-03-21T13:11:54.997497+0000","last_fullsized":"2024-03-21T13:11:54.997497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:54:20.727690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749232+0000","last_change":"2024-03-21T13:11:20.312629+0000","last_active":"2024-03-21T13:11:54.749232+0000","last_peered":"2024-03-21T13:11:54.749232+0000","last_clean":"2024-03-21T13:11:54.749232+0000","last_became_active":"2024-03-21T13:11:20.312314+0000","last_became_peered":"2024-03-21T13:11:20.312314+0000","last_unstale":"2024-03-21T13:11:54.749232+0000","last_undegraded":"2024-03-21T13:11:54.749232+0000","last_fullsized":"2024-03-21T13:11:54.749232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:38:18.330065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856604+0000","last_change":"2024-03-21T13:11:16.151204+0000","last_active":"2024-03-21T13:11:54.856604+0000","last_peered":"2024-03-21T13:11:54.856604+0000","last_clean":"2024-03-21T13:11:54.856604+0000","last_became_active":"2024-03-21T13:11:16.150898+0000","last_became_peered":"2024-03-21T13:11:16.150898+0000","last_unstale":"2024-03-21T13:11:54.856604+0000","last_undegraded":"2024-03-21T13:11:54.856604+0000","last_fullsized":"2024-03-21T13:11:54.856604+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:17:21.887091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998629+0000","last_change":"2024-03-21T13:11:18.296824+0000","last_active":"2024-03-21T13:11:54.998629+0000","last_peered":"2024-03-21T13:11:54.998629+0000","last_clean":"2024-03-21T13:11:54.998629+0000","last_became_active":"2024-03-21T13:11:18.296208+0000","last_became_peered":"2024-03-21T13:11:18.296208+0000","last_unstale":"2024-03-21T13:11:54.998629+0000","last_undegraded":"2024-03-21T13:11:54.998629+0000","last_fullsized":"2024-03-21T13:11:54.998629+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:47:30.252913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746721+0000","last_change":"2024-03-21T13:11:22.329711+0000","last_active":"2024-03-21T13:11:54.746721+0000","last_peered":"2024-03-21T13:11:54.746721+0000","last_clean":"2024-03-21T13:11:54.746721+0000","last_became_active":"2024-03-21T13:11:22.329391+0000","last_became_peered":"2024-03-21T13:11:22.329391+0000","last_unstale":"2024-03-21T13:11:54.746721+0000","last_undegraded":"2024-03-21T13:11:54.746721+0000","last_fullsized":"2024-03-21T13:11:54.746721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:44:57.425497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998540+0000","last_change":"2024-03-21T13:11:20.313852+0000","last_active":"2024-03-21T13:11:54.998540+0000","last_peered":"2024-03-21T13:11:54.998540+0000","last_clean":"2024-03-21T13:11:54.998540+0000","last_became_active":"2024-03-21T13:11:20.312718+0000","last_became_peered":"2024-03-21T13:11:20.312718+0000","last_unstale":"2024-03-21T13:11:54.998540+0000","last_undegraded":"2024-03-21T13:11:54.998540+0000","last_fullsized":"2024-03-21T13:11:54.998540+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:17:43.302267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748570+0000","last_change":"2024-03-21T13:11:16.145627+0000","last_active":"2024-03-21T13:11:54.748570+0000","last_peered":"2024-03-21T13:11:54.748570+0000","last_clean":"2024-03-21T13:11:54.748570+0000","last_became_active":"2024-03-21T13:11:16.145334+0000","last_became_peered":"2024-03-21T13:11:16.145334+0000","last_unstale":"2024-03-21T13:11:54.748570+0000","last_undegraded":"2024-03-21T13:11:54.748570+0000","last_fullsized":"2024-03-21T13:11:54.748570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:21.993340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748709+0000","last_change":"2024-03-21T13:11:18.296351+0000","last_active":"2024-03-21T13:11:54.748709+0000","last_peered":"2024-03-21T13:11:54.748709+0000","last_clean":"2024-03-21T13:11:54.748709+0000","last_became_active":"2024-03-21T13:11:18.294511+0000","last_became_peered":"2024-03-21T13:11:18.294511+0000","last_unstale":"2024-03-21T13:11:54.748709+0000","last_undegraded":"2024-03-21T13:11:54.748709+0000","last_fullsized":"2024-03-21T13:11:54.748709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:29:33.073734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997784+0000","last_change":"2024-03-21T13:11:22.326706+0000","last_active":"2024-03-21T13:11:54.997784+0000","last_peered":"2024-03-21T13:11:54.997784+0000","last_clean":"2024-03-21T13:11:54.997784+0000","last_became_active":"2024-03-21T13:11:22.326379+0000","last_became_peered":"2024-03-21T13:11:22.326379+0000","last_unstale":"2024-03-21T13:11:54.997784+0000","last_undegraded":"2024-03-21T13:11:54.997784+0000","last_fullsized":"2024-03-21T13:11:54.997784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:46:48.406961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210868+0000","last_change":"2024-03-21T13:11:20.317767+0000","last_active":"2024-03-21T13:12:31.210868+0000","last_peered":"2024-03-21T13:12:31.210868+0000","last_clean":"2024-03-21T13:12:31.210868+0000","last_became_active":"2024-03-21T13:11:20.317461+0000","last_became_peered":"2024-03-21T13:11:20.317461+0000","last_unstale":"2024-03-21T13:12:31.210868+0000","last_undegraded":"2024-03-21T13:12:31.210868+0000","last_fullsized":"2024-03-21T13:12:31.210868+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:38.172483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856348+0000","last_change":"2024-03-21T13:11:16.149368+0000","last_active":"2024-03-21T13:11:54.856348+0000","last_peered":"2024-03-21T13:11:54.856348+0000","last_clean":"2024-03-21T13:11:54.856348+0000","last_became_active":"2024-03-21T13:11:16.149083+0000","last_became_peered":"2024-03-21T13:11:16.149083+0000","last_unstale":"2024-03-21T13:11:54.856348+0000","last_undegraded":"2024-03-21T13:11:54.856348+0000","last_fullsized":"2024-03-21T13:11:54.856348+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:54:39.456177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.747157+0000","last_change":"2024-03-21T13:11:18.515660+0000","last_active":"2024-03-21T13:11:54.747157+0000","last_peered":"2024-03-21T13:11:54.747157+0000","last_clean":"2024-03-21T13:11:54.747157+0000","last_became_active":"2024-03-21T13:11:18.515363+0000","last_became_peered":"2024-03-21T13:11:18.515363+0000","last_unstale":"2024-03-21T13:11:54.747157+0000","last_undegraded":"2024-03-21T13:11:54.747157+0000","last_fullsized":"2024-03-21T13:11:54.747157+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:46:50.492693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749541+0000","last_change":"2024-03-21T13:11:22.323487+0000","last_active":"2024-03-21T13:11:54.749541+0000","last_peered":"2024-03-21T13:11:54.749541+0000","last_clean":"2024-03-21T13:11:54.749541+0000","last_became_active":"2024-03-21T13:11:22.323160+0000","last_became_peered":"2024-03-21T13:11:22.323160+0000","last_unstale":"2024-03-21T13:11:54.749541+0000","last_undegraded":"2024-03-21T13:11:54.749541+0000","last_fullsized":"2024-03-21T13:11:54.749541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:11:15.105135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210655+0000","last_change":"2024-03-21T13:11:20.311784+0000","last_active":"2024-03-21T13:12:31.210655+0000","last_peered":"2024-03-21T13:12:31.210655+0000","last_clean":"2024-03-21T13:12:31.210655+0000","last_became_active":"2024-03-21T13:11:20.311363+0000","last_became_peered":"2024-03-21T13:11:20.311363+0000","last_unstale":"2024-03-21T13:12:31.210655+0000","last_undegraded":"2024-03-21T13:12:31.210655+0000","last_fullsized":"2024-03-21T13:12:31.210655+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:55:47.547824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997934+0000","last_change":"2024-03-21T13:11:16.153646+0000","last_active":"2024-03-21T13:11:54.997934+0000","last_peered":"2024-03-21T13:11:54.997934+0000","last_clean":"2024-03-21T13:11:54.997934+0000","last_became_active":"2024-03-21T13:11:16.153323+0000","last_became_peered":"2024-03-21T13:11:16.153323+0000","last_unstale":"2024-03-21T13:11:54.997934+0000","last_undegraded":"2024-03-21T13:11:54.997934+0000","last_fullsized":"2024-03-21T13:11:54.997934+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:21:18.416022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.847028+0000","last_change":"2024-03-21T13:11:18.517861+0000","last_active":"2024-03-21T13:11:54.847028+0000","last_peered":"2024-03-21T13:11:54.847028+0000","last_clean":"2024-03-21T13:11:54.847028+0000","last_became_active":"2024-03-21T13:11:18.517501+0000","last_became_peered":"2024-03-21T13:11:18.517501+0000","last_unstale":"2024-03-21T13:11:54.847028+0000","last_undegraded":"2024-03-21T13:11:54.847028+0000","last_fullsized":"2024-03-21T13:11:54.847028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:08.783284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744442+0000","last_change":"2024-03-21T13:11:22.325223+0000","last_active":"2024-03-21T13:11:54.744442+0000","last_peered":"2024-03-21T13:11:54.744442+0000","last_clean":"2024-03-21T13:11:54.744442+0000","last_became_active":"2024-03-21T13:11:22.324908+0000","last_became_peered":"2024-03-21T13:11:22.324908+0000","last_unstale":"2024-03-21T13:11:54.744442+0000","last_undegraded":"2024-03-21T13:11:54.744442+0000","last_fullsized":"2024-03-21T13:11:54.744442+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:54:12.819611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748818+0000","last_change":"2024-03-21T13:11:20.310958+0000","last_active":"2024-03-21T13:11:54.748818+0000","last_peered":"2024-03-21T13:11:54.748818+0000","last_clean":"2024-03-21T13:11:54.748818+0000","last_became_active":"2024-03-21T13:11:20.310475+0000","last_became_peered":"2024-03-21T13:11:20.310475+0000","last_unstale":"2024-03-21T13:11:54.748818+0000","last_undegraded":"2024-03-21T13:11:54.748818+0000","last_fullsized":"2024-03-21T13:11:54.748818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:58:33.751690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856273+0000","last_change":"2024-03-21T13:11:16.146676+0000","last_active":"2024-03-21T13:11:54.856273+0000","last_peered":"2024-03-21T13:11:54.856273+0000","last_clean":"2024-03-21T13:11:54.856273+0000","last_became_active":"2024-03-21T13:11:16.146370+0000","last_became_peered":"2024-03-21T13:11:16.146370+0000","last_unstale":"2024-03-21T13:11:54.856273+0000","last_undegraded":"2024-03-21T13:11:54.856273+0000","last_fullsized":"2024-03-21T13:11:54.856273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:36.564709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998324+0000","last_change":"2024-03-21T13:11:18.517641+0000","last_active":"2024-03-21T13:11:54.998324+0000","last_peered":"2024-03-21T13:11:54.998324+0000","last_clean":"2024-03-21T13:11:54.998324+0000","last_became_active":"2024-03-21T13:11:18.517359+0000","last_became_peered":"2024-03-21T13:11:18.517359+0000","last_unstale":"2024-03-21T13:11:54.998324+0000","last_undegraded":"2024-03-21T13:11:54.998324+0000","last_fullsized":"2024-03-21T13:11:54.998324+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:31.385588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856558+0000","last_change":"2024-03-21T13:11:22.517376+0000","last_active":"2024-03-21T13:11:54.856558+0000","last_peered":"2024-03-21T13:11:54.856558+0000","last_clean":"2024-03-21T13:11:54.856558+0000","last_became_active":"2024-03-21T13:11:22.517100+0000","last_became_peered":"2024-03-21T13:11:22.517100+0000","last_unstale":"2024-03-21T13:11:54.856558+0000","last_undegraded":"2024-03-21T13:11:54.856558+0000","last_fullsized":"2024-03-21T13:11:54.856558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:55:05.359652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846628+0000","last_change":"2024-03-21T13:11:20.312140+0000","last_active":"2024-03-21T13:11:54.846628+0000","last_peered":"2024-03-21T13:11:54.846628+0000","last_clean":"2024-03-21T13:11:54.846628+0000","last_became_active":"2024-03-21T13:11:20.311830+0000","last_became_peered":"2024-03-21T13:11:20.311830+0000","last_unstale":"2024-03-21T13:11:54.846628+0000","last_undegraded":"2024-03-21T13:11:54.846628+0000","last_fullsized":"2024-03-21T13:11:54.846628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:13:16.882822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997619+0000","last_change":"2024-03-21T13:11:16.147247+0000","last_active":"2024-03-21T13:11:54.997619+0000","last_peered":"2024-03-21T13:11:54.997619+0000","last_clean":"2024-03-21T13:11:54.997619+0000","last_became_active":"2024-03-21T13:11:16.146940+0000","last_became_peered":"2024-03-21T13:11:16.146940+0000","last_unstale":"2024-03-21T13:11:54.997619+0000","last_undegraded":"2024-03-21T13:11:54.997619+0000","last_fullsized":"2024-03-21T13:11:54.997619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:29:54.978928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":38,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749116+0000","last_change":"2024-03-21T13:11:18.515801+0000","last_active":"2024-03-21T13:11:54.749116+0000","last_peered":"2024-03-21T13:11:54.749116+0000","last_clean":"2024-03-21T13:11:54.749116+0000","last_became_active":"2024-03-21T13:11:18.515372+0000","last_became_peered":"2024-03-21T13:11:18.515372+0000","last_unstale":"2024-03-21T13:11:54.749116+0000","last_undegraded":"2024-03-21T13:11:54.749116+0000","last_fullsized":"2024-03-21T13:11:54.749116+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:33:49.386828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846557+0000","last_change":"2024-03-21T13:11:22.517043+0000","last_active":"2024-03-21T13:11:54.846557+0000","last_peered":"2024-03-21T13:11:54.846557+0000","last_clean":"2024-03-21T13:11:54.846557+0000","last_became_active":"2024-03-21T13:11:22.516672+0000","last_became_peered":"2024-03-21T13:11:22.516672+0000","last_unstale":"2024-03-21T13:11:54.846557+0000","last_undegraded":"2024-03-21T13:11:54.846557+0000","last_fullsized":"2024-03-21T13:11:54.846557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:03.433181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856873+0000","last_change":"2024-03-21T13:11:16.149051+0000","last_active":"2024-03-21T13:11:54.856873+0000","last_peered":"2024-03-21T13:11:54.856873+0000","last_clean":"2024-03-21T13:11:54.856873+0000","last_became_active":"2024-03-21T13:11:16.148552+0000","last_became_peered":"2024-03-21T13:11:16.148552+0000","last_unstale":"2024-03-21T13:11:54.856873+0000","last_undegraded":"2024-03-21T13:11:54.856873+0000","last_fullsized":"2024-03-21T13:11:54.856873+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:03:14.897415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846261+0000","last_change":"2024-03-21T13:11:18.518098+0000","last_active":"2024-03-21T13:11:54.846261+0000","last_peered":"2024-03-21T13:11:54.846261+0000","last_clean":"2024-03-21T13:11:54.846261+0000","last_became_active":"2024-03-21T13:11:18.517747+0000","last_became_peered":"2024-03-21T13:11:18.517747+0000","last_unstale":"2024-03-21T13:11:54.846261+0000","last_undegraded":"2024-03-21T13:11:54.846261+0000","last_fullsized":"2024-03-21T13:11:54.846261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:11:46.628125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210841+0000","last_change":"2024-03-21T13:11:20.314260+0000","last_active":"2024-03-21T13:12:31.210841+0000","last_peered":"2024-03-21T13:12:31.210841+0000","last_clean":"2024-03-21T13:12:31.210841+0000","last_became_active":"2024-03-21T13:11:20.313944+0000","last_became_peered":"2024-03-21T13:11:20.313944+0000","last_unstale":"2024-03-21T13:12:31.210841+0000","last_undegraded":"2024-03-21T13:12:31.210841+0000","last_fullsized":"2024-03-21T13:12:31.210841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:18:26.325206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749070+0000","last_change":"2024-03-21T13:11:22.330740+0000","last_active":"2024-03-21T13:11:54.749070+0000","last_peered":"2024-03-21T13:11:54.749070+0000","last_clean":"2024-03-21T13:11:54.749070+0000","last_became_active":"2024-03-21T13:11:22.330243+0000","last_became_peered":"2024-03-21T13:11:22.330243+0000","last_unstale":"2024-03-21T13:11:54.749070+0000","last_undegraded":"2024-03-21T13:11:54.749070+0000","last_fullsized":"2024-03-21T13:11:54.749070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:08:57.495977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856837+0000","last_change":"2024-03-21T13:11:16.147817+0000","last_active":"2024-03-21T13:11:54.856837+0000","last_peered":"2024-03-21T13:11:54.856837+0000","last_clean":"2024-03-21T13:11:54.856837+0000","last_became_active":"2024-03-21T13:11:16.147266+0000","last_became_peered":"2024-03-21T13:11:16.147266+0000","last_unstale":"2024-03-21T13:11:54.856837+0000","last_undegraded":"2024-03-21T13:11:54.856837+0000","last_fullsized":"2024-03-21T13:11:54.856837+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:01:21.067524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":52,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748554+0000","last_change":"2024-03-21T13:11:18.305238+0000","last_active":"2024-03-21T13:11:54.748554+0000","last_peered":"2024-03-21T13:11:54.748554+0000","last_clean":"2024-03-21T13:11:54.748554+0000","last_became_active":"2024-03-21T13:11:18.304939+0000","last_became_peered":"2024-03-21T13:11:18.304939+0000","last_unstale":"2024-03-21T13:11:54.748554+0000","last_undegraded":"2024-03-21T13:11:54.748554+0000","last_fullsized":"2024-03-21T13:11:54.748554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:19:27.422003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.743826+0000","last_change":"2024-03-21T13:11:20.311620+0000","last_active":"2024-03-21T13:11:54.743826+0000","last_peered":"2024-03-21T13:11:54.743826+0000","last_clean":"2024-03-21T13:11:54.743826+0000","last_became_active":"2024-03-21T13:11:20.311207+0000","last_became_peered":"2024-03-21T13:11:20.311207+0000","last_unstale":"2024-03-21T13:11:54.743826+0000","last_undegraded":"2024-03-21T13:11:54.743826+0000","last_fullsized":"2024-03-21T13:11:54.743826+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:47.585866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856080+0000","last_change":"2024-03-21T13:11:22.326591+0000","last_active":"2024-03-21T13:11:54.856080+0000","last_peered":"2024-03-21T13:11:54.856080+0000","last_clean":"2024-03-21T13:11:54.856080+0000","last_became_active":"2024-03-21T13:11:22.326308+0000","last_became_peered":"2024-03-21T13:11:22.326308+0000","last_unstale":"2024-03-21T13:11:54.856080+0000","last_undegraded":"2024-03-21T13:11:54.856080+0000","last_fullsized":"2024-03-21T13:11:54.856080+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:24:24.747360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744051+0000","last_change":"2024-03-21T13:11:20.311279+0000","last_active":"2024-03-21T13:11:54.744051+0000","last_peered":"2024-03-21T13:11:54.744051+0000","last_clean":"2024-03-21T13:11:54.744051+0000","last_became_active":"2024-03-21T13:11:20.310943+0000","last_became_peered":"2024-03-21T13:11:20.310943+0000","last_unstale":"2024-03-21T13:11:54.744051+0000","last_undegraded":"2024-03-21T13:11:54.744051+0000","last_fullsized":"2024-03-21T13:11:54.744051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:19:23.005768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856762+0000","last_change":"2024-03-21T13:11:16.147924+0000","last_active":"2024-03-21T13:11:54.856762+0000","last_peered":"2024-03-21T13:11:54.856762+0000","last_clean":"2024-03-21T13:11:54.856762+0000","last_became_active":"2024-03-21T13:11:16.147620+0000","last_became_peered":"2024-03-21T13:11:16.147620+0000","last_unstale":"2024-03-21T13:11:54.856762+0000","last_undegraded":"2024-03-21T13:11:54.856762+0000","last_fullsized":"2024-03-21T13:11:54.856762+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:46.322538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997835+0000","last_change":"2024-03-21T13:11:18.516046+0000","last_active":"2024-03-21T13:11:54.997835+0000","last_peered":"2024-03-21T13:11:54.997835+0000","last_clean":"2024-03-21T13:11:54.997835+0000","last_became_active":"2024-03-21T13:11:18.515653+0000","last_became_peered":"2024-03-21T13:11:18.515653+0000","last_unstale":"2024-03-21T13:11:54.997835+0000","last_undegraded":"2024-03-21T13:11:54.997835+0000","last_fullsized":"2024-03-21T13:11:54.997835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:26:16.712573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997569+0000","last_change":"2024-03-21T13:11:22.318905+0000","last_active":"2024-03-21T13:11:54.997569+0000","last_peered":"2024-03-21T13:11:54.997569+0000","last_clean":"2024-03-21T13:11:54.997569+0000","last_became_active":"2024-03-21T13:11:22.318593+0000","last_became_peered":"2024-03-21T13:11:22.318593+0000","last_unstale":"2024-03-21T13:11:54.997569+0000","last_undegraded":"2024-03-21T13:11:54.997569+0000","last_fullsized":"2024-03-21T13:11:54.997569+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:39.216740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744594+0000","last_change":"2024-03-21T13:11:20.308775+0000","last_active":"2024-03-21T13:11:54.744594+0000","last_peered":"2024-03-21T13:11:54.744594+0000","last_clean":"2024-03-21T13:11:54.744594+0000","last_became_active":"2024-03-21T13:11:20.308439+0000","last_became_peered":"2024-03-21T13:11:20.308439+0000","last_unstale":"2024-03-21T13:11:54.744594+0000","last_undegraded":"2024-03-21T13:11:54.744594+0000","last_fullsized":"2024-03-21T13:11:54.744594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:34:38.666166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846521+0000","last_change":"2024-03-21T13:11:16.143237+0000","last_active":"2024-03-21T13:11:54.846521+0000","last_peered":"2024-03-21T13:11:54.846521+0000","last_clean":"2024-03-21T13:11:54.846521+0000","last_became_active":"2024-03-21T13:11:16.142895+0000","last_became_peered":"2024-03-21T13:11:16.142895+0000","last_unstale":"2024-03-21T13:11:54.846521+0000","last_undegraded":"2024-03-21T13:11:54.846521+0000","last_fullsized":"2024-03-21T13:11:54.846521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:01:26.896341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998485+0000","last_change":"2024-03-21T13:11:18.298211+0000","last_active":"2024-03-21T13:11:54.998485+0000","last_peered":"2024-03-21T13:11:54.998485+0000","last_clean":"2024-03-21T13:11:54.998485+0000","last_became_active":"2024-03-21T13:11:18.297832+0000","last_became_peered":"2024-03-21T13:11:18.297832+0000","last_unstale":"2024-03-21T13:11:54.998485+0000","last_undegraded":"2024-03-21T13:11:54.998485+0000","last_fullsized":"2024-03-21T13:11:54.998485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:12:01.735751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748142+0000","last_change":"2024-03-21T13:11:22.330144+0000","last_active":"2024-03-21T13:11:54.748142+0000","last_peered":"2024-03-21T13:11:54.748142+0000","last_clean":"2024-03-21T13:11:54.748142+0000","last_became_active":"2024-03-21T13:11:22.329794+0000","last_became_peered":"2024-03-21T13:11:22.329794+0000","last_unstale":"2024-03-21T13:11:54.748142+0000","last_undegraded":"2024-03-21T13:11:54.748142+0000","last_fullsized":"2024-03-21T13:11:54.748142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:59:19.513165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"66'2","reported_seq":42,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846364+0000","last_change":"2024-03-21T13:11:20.312696+0000","last_active":"2024-03-21T13:11:54.846364+0000","last_peered":"2024-03-21T13:11:54.846364+0000","last_clean":"2024-03-21T13:11:54.846364+0000","last_became_active":"2024-03-21T13:11:16.149446+0000","last_became_peered":"2024-03-21T13:11:16.149446+0000","last_unstale":"2024-03-21T13:11:54.846364+0000","last_undegraded":"2024-03-21T13:11:54.846364+0000","last_fullsized":"2024-03-21T13:11:54.846364+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:53:03.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001200118,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748295+0000","last_change":"2024-03-21T13:11:16.145737+0000","last_active":"2024-03-21T13:11:54.748295+0000","last_peered":"2024-03-21T13:11:54.748295+0000","last_clean":"2024-03-21T13:11:54.748295+0000","last_became_active":"2024-03-21T13:11:16.144759+0000","last_became_peered":"2024-03-21T13:11:16.144759+0000","last_unstale":"2024-03-21T13:11:54.748295+0000","last_undegraded":"2024-03-21T13:11:54.748295+0000","last_fullsized":"2024-03-21T13:11:54.748295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:47:37.336001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210646+0000","last_change":"2024-03-21T13:11:18.302318+0000","last_active":"2024-03-21T13:12:31.210646+0000","last_peered":"2024-03-21T13:12:31.210646+0000","last_clean":"2024-03-21T13:12:31.210646+0000","last_became_active":"2024-03-21T13:11:18.302003+0000","last_became_peered":"2024-03-21T13:11:18.302003+0000","last_unstale":"2024-03-21T13:12:31.210646+0000","last_undegraded":"2024-03-21T13:12:31.210646+0000","last_fullsized":"2024-03-21T13:12:31.210646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:34:23.355563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998345+0000","last_change":"2024-03-21T13:11:20.315475+0000","last_active":"2024-03-21T13:11:54.998345+0000","last_peered":"2024-03-21T13:11:54.998345+0000","last_clean":"2024-03-21T13:11:54.998345+0000","last_became_active":"2024-03-21T13:11:20.315107+0000","last_became_peered":"2024-03-21T13:11:20.315107+0000","last_unstale":"2024-03-21T13:11:54.998345+0000","last_undegraded":"2024-03-21T13:11:54.998345+0000","last_fullsized":"2024-03-21T13:11:54.998345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:16:27.491803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856866+0000","last_change":"2024-03-21T13:11:22.516024+0000","last_active":"2024-03-21T13:11:54.856866+0000","last_peered":"2024-03-21T13:11:54.856866+0000","last_clean":"2024-03-21T13:11:54.856866+0000","last_became_active":"2024-03-21T13:11:22.515689+0000","last_became_peered":"2024-03-21T13:11:22.515689+0000","last_unstale":"2024-03-21T13:11:54.856866+0000","last_undegraded":"2024-03-21T13:11:54.856866+0000","last_fullsized":"2024-03-21T13:11:54.856866+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:21:44.792827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998734+0000","last_change":"2024-03-21T13:11:16.151238+0000","last_active":"2024-03-21T13:11:54.998734+0000","last_peered":"2024-03-21T13:11:54.998734+0000","last_clean":"2024-03-21T13:11:54.998734+0000","last_became_active":"2024-03-21T13:11:16.150674+0000","last_became_peered":"2024-03-21T13:11:16.150674+0000","last_unstale":"2024-03-21T13:11:54.998734+0000","last_undegraded":"2024-03-21T13:11:54.998734+0000","last_fullsized":"2024-03-21T13:11:54.998734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:20.741716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.743902+0000","last_change":"2024-03-21T13:11:20.300589+0000","last_active":"2024-03-21T13:11:54.743902+0000","last_peered":"2024-03-21T13:11:54.743902+0000","last_clean":"2024-03-21T13:11:54.743902+0000","last_became_active":"2024-03-21T13:11:16.140637+0000","last_became_peered":"2024-03-21T13:11:16.140637+0000","last_unstale":"2024-03-21T13:11:54.743902+0000","last_undegraded":"2024-03-21T13:11:54.743902+0000","last_fullsized":"2024-03-21T13:11:54.743902+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:59.393367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055733400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999147+0000","last_change":"2024-03-21T13:11:18.297598+0000","last_active":"2024-03-21T13:11:54.999147+0000","last_peered":"2024-03-21T13:11:54.999147+0000","last_clean":"2024-03-21T13:11:54.999147+0000","last_became_active":"2024-03-21T13:11:18.294401+0000","last_became_peered":"2024-03-21T13:11:18.294401+0000","last_unstale":"2024-03-21T13:11:54.999147+0000","last_undegraded":"2024-03-21T13:11:54.999147+0000","last_fullsized":"2024-03-21T13:11:54.999147+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:45:16.818297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744066+0000","last_change":"2024-03-21T13:11:20.312133+0000","last_active":"2024-03-21T13:11:54.744066+0000","last_peered":"2024-03-21T13:11:54.744066+0000","last_clean":"2024-03-21T13:11:54.744066+0000","last_became_active":"2024-03-21T13:11:20.311822+0000","last_became_peered":"2024-03-21T13:11:20.311822+0000","last_unstale":"2024-03-21T13:11:54.744066+0000","last_undegraded":"2024-03-21T13:11:54.744066+0000","last_fullsized":"2024-03-21T13:11:54.744066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:18:05.076077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846685+0000","last_change":"2024-03-21T13:11:22.326599+0000","last_active":"2024-03-21T13:11:54.846685+0000","last_peered":"2024-03-21T13:11:54.846685+0000","last_clean":"2024-03-21T13:11:54.846685+0000","last_became_active":"2024-03-21T13:11:22.326313+0000","last_became_peered":"2024-03-21T13:11:22.326313+0000","last_unstale":"2024-03-21T13:11:54.846685+0000","last_undegraded":"2024-03-21T13:11:54.846685+0000","last_fullsized":"2024-03-21T13:11:54.846685+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:19:24.243393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'128","reported_seq":277,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:56.847154+0000","last_change":"2024-03-21T13:11:06.402763+0000","last_active":"2024-03-21T13:11:56.847154+0000","last_peered":"2024-03-21T13:11:56.847154+0000","last_clean":"2024-03-21T13:11:56.847154+0000","last_became_active":"2024-03-21T13:11:06.402276+0000","last_became_peered":"2024-03-21T13:11:06.402276+0000","last_unstale":"2024-03-21T13:11:56.847154+0000","last_undegraded":"2024-03-21T13:11:56.847154+0000","last_fullsized":"2024-03-21T13:11:56.847154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:01.992315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:09:01.992315+0000","last_clean_scrub_stamp":"2024-03-21T13:09:01.992315+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:09:10.536340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746899+0000","last_change":"2024-03-21T13:11:16.149764+0000","last_active":"2024-03-21T13:11:54.746899+0000","last_peered":"2024-03-21T13:11:54.746899+0000","last_clean":"2024-03-21T13:11:54.746899+0000","last_became_active":"2024-03-21T13:11:16.149486+0000","last_became_peered":"2024-03-21T13:11:16.149486+0000","last_unstale":"2024-03-21T13:11:54.746899+0000","last_undegraded":"2024-03-21T13:11:54.746899+0000","last_fullsized":"2024-03-21T13:11:54.746899+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:27:45.992677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":68,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210831+0000","last_change":"2024-03-21T13:11:18.516031+0000","last_active":"2024-03-21T13:12:31.210831+0000","last_peered":"2024-03-21T13:12:31.210831+0000","last_clean":"2024-03-21T13:12:31.210831+0000","last_became_active":"2024-03-21T13:11:18.515668+0000","last_became_peered":"2024-03-21T13:11:18.515668+0000","last_unstale":"2024-03-21T13:12:31.210831+0000","last_undegraded":"2024-03-21T13:12:31.210831+0000","last_fullsized":"2024-03-21T13:12:31.210831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:01:27.045136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856622+0000","last_change":"2024-03-21T13:11:20.319665+0000","last_active":"2024-03-21T13:11:54.856622+0000","last_peered":"2024-03-21T13:11:54.856622+0000","last_clean":"2024-03-21T13:11:54.856622+0000","last_became_active":"2024-03-21T13:11:20.319327+0000","last_became_peered":"2024-03-21T13:11:20.319327+0000","last_unstale":"2024-03-21T13:11:54.856622+0000","last_undegraded":"2024-03-21T13:11:54.856622+0000","last_fullsized":"2024-03-21T13:11:54.856622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:56:08.644596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998615+0000","last_change":"2024-03-21T13:11:22.326840+0000","last_active":"2024-03-21T13:11:54.998615+0000","last_peered":"2024-03-21T13:11:54.998615+0000","last_clean":"2024-03-21T13:11:54.998615+0000","last_became_active":"2024-03-21T13:11:22.326549+0000","last_became_peered":"2024-03-21T13:11:22.326549+0000","last_unstale":"2024-03-21T13:11:54.998615+0000","last_undegraded":"2024-03-21T13:11:54.998615+0000","last_fullsized":"2024-03-21T13:11:54.998615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:10:53.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997737+0000","last_change":"2024-03-21T13:11:16.153450+0000","last_active":"2024-03-21T13:11:54.997737+0000","last_peered":"2024-03-21T13:11:54.997737+0000","last_clean":"2024-03-21T13:11:54.997737+0000","last_became_active":"2024-03-21T13:11:16.153127+0000","last_became_peered":"2024-03-21T13:11:16.153127+0000","last_unstale":"2024-03-21T13:11:54.997737+0000","last_undegraded":"2024-03-21T13:11:54.997737+0000","last_fullsized":"2024-03-21T13:11:54.997737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:39:49.994223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":122,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:39.730320+0000","last_change":"2024-03-21T13:11:20.307323+0000","last_active":"2024-03-21T13:12:39.730320+0000","last_peered":"2024-03-21T13:12:39.730320+0000","last_clean":"2024-03-21T13:12:39.730320+0000","last_became_active":"2024-03-21T13:11:16.147728+0000","last_became_peered":"2024-03-21T13:11:16.147728+0000","last_unstale":"2024-03-21T13:12:39.730320+0000","last_undegraded":"2024-03-21T13:12:39.730320+0000","last_fullsized":"2024-03-21T13:12:39.730320+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:35:29.397384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046438600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":47,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997888+0000","last_change":"2024-03-21T13:11:18.295445+0000","last_active":"2024-03-21T13:11:54.997888+0000","last_peered":"2024-03-21T13:11:54.997888+0000","last_clean":"2024-03-21T13:11:54.997888+0000","last_became_active":"2024-03-21T13:11:18.295126+0000","last_became_peered":"2024-03-21T13:11:18.295126+0000","last_unstale":"2024-03-21T13:11:54.997888+0000","last_undegraded":"2024-03-21T13:11:54.997888+0000","last_fullsized":"2024-03-21T13:11:54.997888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:55.259101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846952+0000","last_change":"2024-03-21T13:11:20.319609+0000","last_active":"2024-03-21T13:11:54.846952+0000","last_peered":"2024-03-21T13:11:54.846952+0000","last_clean":"2024-03-21T13:11:54.846952+0000","last_became_active":"2024-03-21T13:11:20.319321+0000","last_became_peered":"2024-03-21T13:11:20.319321+0000","last_unstale":"2024-03-21T13:11:54.846952+0000","last_undegraded":"2024-03-21T13:11:54.846952+0000","last_fullsized":"2024-03-21T13:11:54.846952+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:51:41.656959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997939+0000","last_change":"2024-03-21T13:11:22.517524+0000","last_active":"2024-03-21T13:11:54.997939+0000","last_peered":"2024-03-21T13:11:54.997939+0000","last_clean":"2024-03-21T13:11:54.997939+0000","last_became_active":"2024-03-21T13:11:22.517207+0000","last_became_peered":"2024-03-21T13:11:22.517207+0000","last_unstale":"2024-03-21T13:11:54.997939+0000","last_undegraded":"2024-03-21T13:11:54.997939+0000","last_fullsized":"2024-03-21T13:11:54.997939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:12.083567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998836+0000","last_change":"2024-03-21T13:11:16.149446+0000","last_active":"2024-03-21T13:11:54.998836+0000","last_peered":"2024-03-21T13:11:54.998836+0000","last_clean":"2024-03-21T13:11:54.998836+0000","last_became_active":"2024-03-21T13:11:16.149156+0000","last_became_peered":"2024-03-21T13:11:16.149156+0000","last_unstale":"2024-03-21T13:11:54.998836+0000","last_undegraded":"2024-03-21T13:11:54.998836+0000","last_fullsized":"2024-03-21T13:11:54.998836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:30:23.099060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":65,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998397+0000","last_change":"2024-03-21T13:11:18.515699+0000","last_active":"2024-03-21T13:11:54.998397+0000","last_peered":"2024-03-21T13:11:54.998397+0000","last_clean":"2024-03-21T13:11:54.998397+0000","last_became_active":"2024-03-21T13:11:18.515414+0000","last_became_peered":"2024-03-21T13:11:18.515414+0000","last_unstale":"2024-03-21T13:11:54.998397+0000","last_undegraded":"2024-03-21T13:11:54.998397+0000","last_fullsized":"2024-03-21T13:11:54.998397+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:01.179571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997832+0000","last_change":"2024-03-21T13:11:20.311707+0000","last_active":"2024-03-21T13:11:54.997832+0000","last_peered":"2024-03-21T13:11:54.997832+0000","last_clean":"2024-03-21T13:11:54.997832+0000","last_became_active":"2024-03-21T13:11:20.311297+0000","last_became_peered":"2024-03-21T13:11:20.311297+0000","last_unstale":"2024-03-21T13:11:54.997832+0000","last_undegraded":"2024-03-21T13:11:54.997832+0000","last_fullsized":"2024-03-21T13:11:54.997832+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:32:40.830229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997497+0000","last_change":"2024-03-21T13:11:22.327442+0000","last_active":"2024-03-21T13:11:54.997497+0000","last_peered":"2024-03-21T13:11:54.997497+0000","last_clean":"2024-03-21T13:11:54.997497+0000","last_became_active":"2024-03-21T13:11:22.327159+0000","last_became_peered":"2024-03-21T13:11:22.327159+0000","last_unstale":"2024-03-21T13:11:54.997497+0000","last_undegraded":"2024-03-21T13:11:54.997497+0000","last_fullsized":"2024-03-21T13:11:54.997497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:51.930027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846915+0000","last_change":"2024-03-21T13:11:16.140879+0000","last_active":"2024-03-21T13:11:54.846915+0000","last_peered":"2024-03-21T13:11:54.846915+0000","last_clean":"2024-03-21T13:11:54.846915+0000","last_became_active":"2024-03-21T13:11:16.140580+0000","last_became_peered":"2024-03-21T13:11:16.140580+0000","last_unstale":"2024-03-21T13:11:54.846915+0000","last_undegraded":"2024-03-21T13:11:54.846915+0000","last_fullsized":"2024-03-21T13:11:54.846915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:22:10.317407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.743875+0000","last_change":"2024-03-21T13:11:44.606939+0000","last_active":"2024-03-21T13:11:54.743875+0000","last_peered":"2024-03-21T13:11:54.743875+0000","last_clean":"2024-03-21T13:11:54.743875+0000","last_became_active":"2024-03-21T13:11:44.593687+0000","last_became_peered":"2024-03-21T13:11:44.593687+0000","last_unstale":"2024-03-21T13:11:54.743875+0000","last_undegraded":"2024-03-21T13:11:54.743875+0000","last_fullsized":"2024-03-21T13:11:54.743875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:13:04.981768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210645+0000","last_change":"2024-03-21T13:11:20.316976+0000","last_active":"2024-03-21T13:12:31.210645+0000","last_peered":"2024-03-21T13:12:31.210645+0000","last_clean":"2024-03-21T13:12:31.210645+0000","last_became_active":"2024-03-21T13:11:20.316676+0000","last_became_peered":"2024-03-21T13:11:20.316676+0000","last_unstale":"2024-03-21T13:12:31.210645+0000","last_undegraded":"2024-03-21T13:12:31.210645+0000","last_fullsized":"2024-03-21T13:12:31.210645+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:59:39.772223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856631+0000","last_change":"2024-03-21T13:11:22.518842+0000","last_active":"2024-03-21T13:11:54.856631+0000","last_peered":"2024-03-21T13:11:54.856631+0000","last_clean":"2024-03-21T13:11:54.856631+0000","last_became_active":"2024-03-21T13:11:22.518396+0000","last_became_peered":"2024-03-21T13:11:22.518396+0000","last_unstale":"2024-03-21T13:11:54.856631+0000","last_undegraded":"2024-03-21T13:11:54.856631+0000","last_fullsized":"2024-03-21T13:11:54.856631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:24:37.971321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998078+0000","last_change":"2024-03-21T13:11:16.147549+0000","last_active":"2024-03-21T13:11:54.998078+0000","last_peered":"2024-03-21T13:11:54.998078+0000","last_clean":"2024-03-21T13:11:54.998078+0000","last_became_active":"2024-03-21T13:11:16.147232+0000","last_became_peered":"2024-03-21T13:11:16.147232+0000","last_unstale":"2024-03-21T13:11:54.998078+0000","last_undegraded":"2024-03-21T13:11:54.998078+0000","last_fullsized":"2024-03-21T13:11:54.998078+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:11.121576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748422+0000","last_change":"2024-03-21T13:11:18.296211+0000","last_active":"2024-03-21T13:11:54.748422+0000","last_peered":"2024-03-21T13:11:54.748422+0000","last_clean":"2024-03-21T13:11:54.748422+0000","last_became_active":"2024-03-21T13:11:18.295886+0000","last_became_peered":"2024-03-21T13:11:18.295886+0000","last_unstale":"2024-03-21T13:11:54.748422+0000","last_undegraded":"2024-03-21T13:11:54.748422+0000","last_fullsized":"2024-03-21T13:11:54.748422+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:50:18.908779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210657+0000","last_change":"2024-03-21T13:11:20.307730+0000","last_active":"2024-03-21T13:12:31.210657+0000","last_peered":"2024-03-21T13:12:31.210657+0000","last_clean":"2024-03-21T13:12:31.210657+0000","last_became_active":"2024-03-21T13:11:20.307389+0000","last_became_peered":"2024-03-21T13:11:20.307389+0000","last_unstale":"2024-03-21T13:12:31.210657+0000","last_undegraded":"2024-03-21T13:12:31.210657+0000","last_fullsized":"2024-03-21T13:12:31.210657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:49:05.193349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749357+0000","last_change":"2024-03-21T13:11:22.327360+0000","last_active":"2024-03-21T13:11:54.749357+0000","last_peered":"2024-03-21T13:11:54.749357+0000","last_clean":"2024-03-21T13:11:54.749357+0000","last_became_active":"2024-03-21T13:11:22.326719+0000","last_became_peered":"2024-03-21T13:11:22.326719+0000","last_unstale":"2024-03-21T13:11:54.749357+0000","last_undegraded":"2024-03-21T13:11:54.749357+0000","last_fullsized":"2024-03-21T13:11:54.749357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:40:30.105040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748618+0000","last_change":"2024-03-21T13:11:16.152317+0000","last_active":"2024-03-21T13:11:54.748618+0000","last_peered":"2024-03-21T13:11:54.748618+0000","last_clean":"2024-03-21T13:11:54.748618+0000","last_became_active":"2024-03-21T13:11:16.152039+0000","last_became_peered":"2024-03-21T13:11:16.152039+0000","last_unstale":"2024-03-21T13:11:54.748618+0000","last_undegraded":"2024-03-21T13:11:54.748618+0000","last_fullsized":"2024-03-21T13:11:54.748618+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:09:33.839419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":61,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746349+0000","last_change":"2024-03-21T13:11:18.515974+0000","last_active":"2024-03-21T13:11:54.746349+0000","last_peered":"2024-03-21T13:11:54.746349+0000","last_clean":"2024-03-21T13:11:54.746349+0000","last_became_active":"2024-03-21T13:11:18.515526+0000","last_became_peered":"2024-03-21T13:11:18.515526+0000","last_unstale":"2024-03-21T13:11:54.746349+0000","last_undegraded":"2024-03-21T13:11:54.746349+0000","last_fullsized":"2024-03-21T13:11:54.746349+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:23:55.127766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748108+0000","last_change":"2024-03-21T13:11:20.311803+0000","last_active":"2024-03-21T13:11:54.748108+0000","last_peered":"2024-03-21T13:11:54.748108+0000","last_clean":"2024-03-21T13:11:54.748108+0000","last_became_active":"2024-03-21T13:11:20.311467+0000","last_became_peered":"2024-03-21T13:11:20.311467+0000","last_unstale":"2024-03-21T13:11:54.748108+0000","last_undegraded":"2024-03-21T13:11:54.748108+0000","last_fullsized":"2024-03-21T13:11:54.748108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:41.260619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846824+0000","last_change":"2024-03-21T13:11:22.319485+0000","last_active":"2024-03-21T13:11:54.846824+0000","last_peered":"2024-03-21T13:11:54.846824+0000","last_clean":"2024-03-21T13:11:54.846824+0000","last_became_active":"2024-03-21T13:11:22.319039+0000","last_became_peered":"2024-03-21T13:11:22.319039+0000","last_unstale":"2024-03-21T13:11:54.846824+0000","last_undegraded":"2024-03-21T13:11:54.846824+0000","last_fullsized":"2024-03-21T13:11:54.846824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:22:08.693850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.211003+0000","last_change":"2024-03-21T13:11:20.314087+0000","last_active":"2024-03-21T13:12:31.211003+0000","last_peered":"2024-03-21T13:12:31.211003+0000","last_clean":"2024-03-21T13:12:31.211003+0000","last_became_active":"2024-03-21T13:11:20.313552+0000","last_became_peered":"2024-03-21T13:11:20.313552+0000","last_unstale":"2024-03-21T13:12:31.211003+0000","last_undegraded":"2024-03-21T13:12:31.211003+0000","last_fullsized":"2024-03-21T13:12:31.211003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:23:54.518787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746263+0000","last_change":"2024-03-21T13:11:16.148240+0000","last_active":"2024-03-21T13:11:54.746263+0000","last_peered":"2024-03-21T13:11:54.746263+0000","last_clean":"2024-03-21T13:11:54.746263+0000","last_became_active":"2024-03-21T13:11:16.147954+0000","last_became_peered":"2024-03-21T13:11:16.147954+0000","last_unstale":"2024-03-21T13:11:54.746263+0000","last_undegraded":"2024-03-21T13:11:54.746263+0000","last_fullsized":"2024-03-21T13:11:54.746263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:12:27.646021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999242+0000","last_change":"2024-03-21T13:11:18.302393+0000","last_active":"2024-03-21T13:11:54.999242+0000","last_peered":"2024-03-21T13:11:54.999242+0000","last_clean":"2024-03-21T13:11:54.999242+0000","last_became_active":"2024-03-21T13:11:18.302060+0000","last_became_peered":"2024-03-21T13:11:18.302060+0000","last_unstale":"2024-03-21T13:11:54.999242+0000","last_undegraded":"2024-03-21T13:11:54.999242+0000","last_fullsized":"2024-03-21T13:11:54.999242+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:31:26.456952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.747072+0000","last_change":"2024-03-21T13:11:22.516838+0000","last_active":"2024-03-21T13:11:54.747072+0000","last_peered":"2024-03-21T13:11:54.747072+0000","last_clean":"2024-03-21T13:11:54.747072+0000","last_became_active":"2024-03-21T13:11:22.516417+0000","last_became_peered":"2024-03-21T13:11:22.516417+0000","last_unstale":"2024-03-21T13:11:54.747072+0000","last_undegraded":"2024-03-21T13:11:54.747072+0000","last_fullsized":"2024-03-21T13:11:54.747072+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:47:12.712391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846897+0000","last_change":"2024-03-21T13:11:20.318949+0000","last_active":"2024-03-21T13:11:54.846897+0000","last_peered":"2024-03-21T13:11:54.846897+0000","last_clean":"2024-03-21T13:11:54.846897+0000","last_became_active":"2024-03-21T13:11:20.318618+0000","last_became_peered":"2024-03-21T13:11:20.318618+0000","last_unstale":"2024-03-21T13:11:54.846897+0000","last_undegraded":"2024-03-21T13:11:54.846897+0000","last_fullsized":"2024-03-21T13:11:54.846897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:36:11.784874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748148+0000","last_change":"2024-03-21T13:11:16.143852+0000","last_active":"2024-03-21T13:11:54.748148+0000","last_peered":"2024-03-21T13:11:54.748148+0000","last_clean":"2024-03-21T13:11:54.748148+0000","last_became_active":"2024-03-21T13:11:16.143563+0000","last_became_peered":"2024-03-21T13:11:16.143563+0000","last_unstale":"2024-03-21T13:11:54.748148+0000","last_undegraded":"2024-03-21T13:11:54.748148+0000","last_fullsized":"2024-03-21T13:11:54.748148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:39:24.221692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748970+0000","last_change":"2024-03-21T13:11:18.295269+0000","last_active":"2024-03-21T13:11:54.748970+0000","last_peered":"2024-03-21T13:11:54.748970+0000","last_clean":"2024-03-21T13:11:54.748970+0000","last_became_active":"2024-03-21T13:11:18.294939+0000","last_became_peered":"2024-03-21T13:11:18.294939+0000","last_unstale":"2024-03-21T13:11:54.748970+0000","last_undegraded":"2024-03-21T13:11:54.748970+0000","last_fullsized":"2024-03-21T13:11:54.748970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:23:44.006873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744375+0000","last_change":"2024-03-21T13:11:22.326202+0000","last_active":"2024-03-21T13:11:54.744375+0000","last_peered":"2024-03-21T13:11:54.744375+0000","last_clean":"2024-03-21T13:11:54.744375+0000","last_became_active":"2024-03-21T13:11:22.325880+0000","last_became_peered":"2024-03-21T13:11:22.325880+0000","last_unstale":"2024-03-21T13:11:54.744375+0000","last_undegraded":"2024-03-21T13:11:54.744375+0000","last_fullsized":"2024-03-21T13:11:54.744375+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:44:37.952019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999111+0000","last_change":"2024-03-21T13:11:20.311104+0000","last_active":"2024-03-21T13:11:54.999111+0000","last_peered":"2024-03-21T13:11:54.999111+0000","last_clean":"2024-03-21T13:11:54.999111+0000","last_became_active":"2024-03-21T13:11:20.310781+0000","last_became_peered":"2024-03-21T13:11:20.310781+0000","last_unstale":"2024-03-21T13:11:54.999111+0000","last_undegraded":"2024-03-21T13:11:54.999111+0000","last_fullsized":"2024-03-21T13:11:54.999111+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:20:42.727931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997693+0000","last_change":"2024-03-21T13:11:16.149568+0000","last_active":"2024-03-21T13:11:54.997693+0000","last_peered":"2024-03-21T13:11:54.997693+0000","last_clean":"2024-03-21T13:11:54.997693+0000","last_became_active":"2024-03-21T13:11:16.149289+0000","last_became_peered":"2024-03-21T13:11:16.149289+0000","last_unstale":"2024-03-21T13:11:54.997693+0000","last_undegraded":"2024-03-21T13:11:54.997693+0000","last_fullsized":"2024-03-21T13:11:54.997693+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:50:12.149327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":81,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749036+0000","last_change":"2024-03-21T13:11:44.601154+0000","last_active":"2024-03-21T13:11:54.749036+0000","last_peered":"2024-03-21T13:11:54.749036+0000","last_clean":"2024-03-21T13:11:54.749036+0000","last_became_active":"2024-03-21T13:11:44.592315+0000","last_became_peered":"2024-03-21T13:11:44.592315+0000","last_unstale":"2024-03-21T13:11:54.749036+0000","last_undegraded":"2024-03-21T13:11:54.749036+0000","last_fullsized":"2024-03-21T13:11:54.749036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:12:31.323995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749452+0000","last_change":"2024-03-21T13:11:22.325527+0000","last_active":"2024-03-21T13:11:54.749452+0000","last_peered":"2024-03-21T13:11:54.749452+0000","last_clean":"2024-03-21T13:11:54.749452+0000","last_became_active":"2024-03-21T13:11:22.325174+0000","last_became_peered":"2024-03-21T13:11:22.325174+0000","last_unstale":"2024-03-21T13:11:54.749452+0000","last_undegraded":"2024-03-21T13:11:54.749452+0000","last_fullsized":"2024-03-21T13:11:54.749452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:20:43.595688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210657+0000","last_change":"2024-03-21T13:11:20.314517+0000","last_active":"2024-03-21T13:12:31.210657+0000","last_peered":"2024-03-21T13:12:31.210657+0000","last_clean":"2024-03-21T13:12:31.210657+0000","last_became_active":"2024-03-21T13:11:20.314190+0000","last_became_peered":"2024-03-21T13:11:20.314190+0000","last_unstale":"2024-03-21T13:12:31.210657+0000","last_undegraded":"2024-03-21T13:12:31.210657+0000","last_fullsized":"2024-03-21T13:12:31.210657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:57:37.488059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749239+0000","last_change":"2024-03-21T13:11:16.153355+0000","last_active":"2024-03-21T13:11:54.749239+0000","last_peered":"2024-03-21T13:11:54.749239+0000","last_clean":"2024-03-21T13:11:54.749239+0000","last_became_active":"2024-03-21T13:11:16.152983+0000","last_became_peered":"2024-03-21T13:11:16.152983+0000","last_unstale":"2024-03-21T13:11:54.749239+0000","last_undegraded":"2024-03-21T13:11:54.749239+0000","last_fullsized":"2024-03-21T13:11:54.749239+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:14:31.204784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748395+0000","last_change":"2024-03-21T13:11:18.300886+0000","last_active":"2024-03-21T13:11:54.748395+0000","last_peered":"2024-03-21T13:11:54.748395+0000","last_clean":"2024-03-21T13:11:54.748395+0000","last_became_active":"2024-03-21T13:11:18.300491+0000","last_became_peered":"2024-03-21T13:11:18.300491+0000","last_unstale":"2024-03-21T13:11:54.748395+0000","last_undegraded":"2024-03-21T13:11:54.748395+0000","last_fullsized":"2024-03-21T13:11:54.748395+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:58:19.420711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168598,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":931211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430235,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712620,"statfs":{"total":95995035648,"available":95961722880,"internally_reserved":0,"allocated":1384448,"data_stored":928812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":851968,"data_stored":340022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":930042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":339580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":827392,"data_stored":339335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":831488,"data_stored":340409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691872,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":802816,"data_stored":338893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:12:41.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T13:12:41.900 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:42.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:42 smithi089 ceph-mon[30095]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:42.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:42 smithi089 ceph-mon[30095]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:42.272 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:42 smithi089 ceph-mon[39209]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:42.272 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:42 smithi089 ceph-mon[39209]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:42 smithi152 ceph-mon[34051]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:42 smithi152 ceph-mon[34051]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:43.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.372+0000 7f4fe14ba700 1 -- 172.21.15.89:0/251834934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc102ba0 msgr2=0x7f4fdc10f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:43.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.372+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/251834934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc102ba0 0x7f4fdc10f040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009f90 tx=0x7f4fd400b330 comp rx=0 tx=0).stop 2024-03-21T13:12:43.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 -- 172.21.15.89:0/251834934 shutdown_connections 2024-03-21T13:12:43.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/251834934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc102ba0 0x7f4fdc10f040 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/251834934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fdc108290 0x7f4fdc108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/251834934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fdc1021e0 0x7f4fdc102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 -- 172.21.15.89:0/251834934 >> 172.21.15.89:0/251834934 conn(0x7f4fdc0792c0 msgr2=0x7f4fdc0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 -- 172.21.15.89:0/251834934 shutdown_connections 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 -- 172.21.15.89:0/251834934 wait complete. 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 Processor -- start 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 -- start start 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.373+0000 7f4fe14ba700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc1021e0 0x7f4fdc074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fe14ba700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fdc102ba0 0x7f4fdc0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fe14ba700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fdc108290 0x7f4fdc074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:43.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fe14ba700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc111990 con 0x7f4fdc102ba0 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fe14ba700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc111810 con 0x7f4fdc108290 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fe14ba700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc111b10 con 0x7f4fdc1021e0 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fdaffd700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc1021e0 0x7f4fdc074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fda7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fdc102ba0 0x7f4fdc0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fdaffd700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc1021e0 0x7f4fdc074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:45838/0 (socket says 172.21.15.89:45838) 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fdaffd700 1 -- 172.21.15.89:0/2005372380 learned_addr learned my addr 172.21.15.89:0/2005372380 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fdb7fe700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fdc108290 0x7f4fdc074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fda7fc700 1 -- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc1021e0 msgr2=0x7f4fdc074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fda7fc700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc1021e0 0x7f4fdc074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fda7fc700 1 -- 172.21.15.89:0/2005372380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fdc108290 msgr2=0x7f4fdc074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fda7fc700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fdc108290 0x7f4fdc074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fda7fc700 1 -- 172.21.15.89:0/2005372380 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc1a63b0 con 0x7f4fdc102ba0 2024-03-21T13:12:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fdaffd700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc1021e0 0x7f4fdc074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:12:43.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fda7fc700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fdc102ba0 0x7f4fdc0749d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000eb90 tx=0x7f4fd000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:43.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fc3fff700 1 -- 172.21.15.89:0/2005372380 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000cc90 con 0x7f4fdc102ba0 2024-03-21T13:12:43.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc1a66a0 con 0x7f4fdc102ba0 2024-03-21T13:12:43.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fc3fff700 1 -- 172.21.15.89:0/2005372380 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4fd000ce30 con 0x7f4fdc102ba0 2024-03-21T13:12:43.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fc3fff700 1 -- 172.21.15.89:0/2005372380 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0003e00 con 0x7f4fdc102ba0 2024-03-21T13:12:43.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.374+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc1a6c30 con 0x7f4fdc102ba0 2024-03-21T13:12:43.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.375+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc068e50 con 0x7f4fdc102ba0 2024-03-21T13:12:43.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.376+0000 7f4fc3fff700 1 -- 172.21.15.89:0/2005372380 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f4fd0003fa0 con 0x7f4fdc102ba0 2024-03-21T13:12:43.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.376+0000 7f4fc3fff700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4fc406e340 0x7f4fc4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:43.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.376+0000 7f4fc3fff700 1 -- 172.21.15.89:0/2005372380 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f4fd008eba0 con 0x7f4fdc102ba0 2024-03-21T13:12:43.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.376+0000 7f4fdaffd700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4fc406e340 0x7f4fc4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:43.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.377+0000 7f4fdaffd700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4fc406e340 0x7f4fc4070800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009ea0 tx=0x7f4fcc009290 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:43.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.381+0000 7f4fc3fff700 1 -- 172.21.15.89:0/2005372380 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4fd005be10 con 0x7f4fdc102ba0 2024-03-21T13:12:43.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.573+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 --> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4fdc1074f0 con 0x7f4fc406e340 2024-03-21T13:12:43.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.583+0000 7f4fc3fff700 1 -- 172.21.15.89:0/2005372380 <== mgr.24458 v2:172.21.15.89:6800/2642177467 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346467 (secure 0 0 0) 0x7f4fdc1074f0 con 0x7f4fc406e340 2024-03-21T13:12:43.588 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4fc406e340 msgr2=0x7f4fc4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4fc406e340 0x7f4fc4070800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009ea0 tx=0x7f4fcc009290 comp rx=0 tx=0).stop 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fdc102ba0 msgr2=0x7f4fdc0749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fdc102ba0 0x7f4fdc0749d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000eb90 tx=0x7f4fd000eea0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 shutdown_connections 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fdc108290 0x7f4fdc074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4fc406e340 0x7f4fc4070800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4fdc1021e0 0x7f4fdc074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 --2- 172.21.15.89:0/2005372380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fdc102ba0 0x7f4fdc0749d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.586+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 >> 172.21.15.89:0/2005372380 conn(0x7f4fdc0792c0 msgr2=0x7f4fdc105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:43.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.587+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 shutdown_connections 2024-03-21T13:12:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:43.587+0000 7f4fe14ba700 1 -- 172.21.15.89:0/2005372380 wait complete. 2024-03-21T13:12:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-03-21T13:12:44.329 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-21T13:12:42.764331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1013,"num_read_kb":859,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":257556,"kb_used_data":8124,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":749703660,"statfs":{"total":767960285184,"available":767696547840,"internally_reserved":0,"allocated":8318976,"data_stored":4488304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002669"},"pg_stats":[{"pgid":"6.1a","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749249+0000","last_change":"2024-03-21T13:11:22.327003+0000","last_active":"2024-03-21T13:11:54.749249+0000","last_peered":"2024-03-21T13:11:54.749249+0000","last_clean":"2024-03-21T13:11:54.749249+0000","last_became_active":"2024-03-21T13:11:22.325840+0000","last_became_peered":"2024-03-21T13:11:22.325840+0000","last_unstale":"2024-03-21T13:11:54.749249+0000","last_undegraded":"2024-03-21T13:11:54.749249+0000","last_fullsized":"2024-03-21T13:11:54.749249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:26:27.565490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997987+0000","last_change":"2024-03-21T13:11:20.315075+0000","last_active":"2024-03-21T13:11:54.997987+0000","last_peered":"2024-03-21T13:11:54.997987+0000","last_clean":"2024-03-21T13:11:54.997987+0000","last_became_active":"2024-03-21T13:11:20.314208+0000","last_became_peered":"2024-03-21T13:11:20.314208+0000","last_unstale":"2024-03-21T13:11:54.997987+0000","last_undegraded":"2024-03-21T13:11:54.997987+0000","last_fullsized":"2024-03-21T13:11:54.997987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:40:57.353281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998664+0000","last_change":"2024-03-21T13:11:16.144659+0000","last_active":"2024-03-21T13:11:54.998664+0000","last_peered":"2024-03-21T13:11:54.998664+0000","last_clean":"2024-03-21T13:11:54.998664+0000","last_became_active":"2024-03-21T13:11:16.144369+0000","last_became_peered":"2024-03-21T13:11:16.144369+0000","last_unstale":"2024-03-21T13:11:54.998664+0000","last_undegraded":"2024-03-21T13:11:54.998664+0000","last_fullsized":"2024-03-21T13:11:54.998664+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:16:46.294733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748084+0000","last_change":"2024-03-21T13:11:18.294733+0000","last_active":"2024-03-21T13:11:54.748084+0000","last_peered":"2024-03-21T13:11:54.748084+0000","last_clean":"2024-03-21T13:11:54.748084+0000","last_became_active":"2024-03-21T13:11:18.294374+0000","last_became_peered":"2024-03-21T13:11:18.294374+0000","last_unstale":"2024-03-21T13:11:54.748084+0000","last_undegraded":"2024-03-21T13:11:54.748084+0000","last_fullsized":"2024-03-21T13:11:54.748084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:45:22.942768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749106+0000","last_change":"2024-03-21T13:11:22.518891+0000","last_active":"2024-03-21T13:11:54.749106+0000","last_peered":"2024-03-21T13:11:54.749106+0000","last_clean":"2024-03-21T13:11:54.749106+0000","last_became_active":"2024-03-21T13:11:22.518517+0000","last_became_peered":"2024-03-21T13:11:22.518517+0000","last_unstale":"2024-03-21T13:11:54.749106+0000","last_undegraded":"2024-03-21T13:11:54.749106+0000","last_fullsized":"2024-03-21T13:11:54.749106+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:13:41.048630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748680+0000","last_change":"2024-03-21T13:11:20.312628+0000","last_active":"2024-03-21T13:11:54.748680+0000","last_peered":"2024-03-21T13:11:54.748680+0000","last_clean":"2024-03-21T13:11:54.748680+0000","last_became_active":"2024-03-21T13:11:20.312336+0000","last_became_peered":"2024-03-21T13:11:20.312336+0000","last_unstale":"2024-03-21T13:11:54.748680+0000","last_undegraded":"2024-03-21T13:11:54.748680+0000","last_fullsized":"2024-03-21T13:11:54.748680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:27:15.221560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748402+0000","last_change":"2024-03-21T13:11:16.146703+0000","last_active":"2024-03-21T13:11:54.748402+0000","last_peered":"2024-03-21T13:11:54.748402+0000","last_clean":"2024-03-21T13:11:54.748402+0000","last_became_active":"2024-03-21T13:11:16.146389+0000","last_became_peered":"2024-03-21T13:11:16.146389+0000","last_unstale":"2024-03-21T13:11:54.748402+0000","last_undegraded":"2024-03-21T13:11:54.748402+0000","last_fullsized":"2024-03-21T13:11:54.748402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:44:59.580339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748790+0000","last_change":"2024-03-21T13:11:18.303525+0000","last_active":"2024-03-21T13:11:54.748790+0000","last_peered":"2024-03-21T13:11:54.748790+0000","last_clean":"2024-03-21T13:11:54.748790+0000","last_became_active":"2024-03-21T13:11:18.303103+0000","last_became_peered":"2024-03-21T13:11:18.303103+0000","last_unstale":"2024-03-21T13:11:54.748790+0000","last_undegraded":"2024-03-21T13:11:54.748790+0000","last_fullsized":"2024-03-21T13:11:54.748790+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:18:41.779791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997858+0000","last_change":"2024-03-21T13:11:22.319817+0000","last_active":"2024-03-21T13:11:54.997858+0000","last_peered":"2024-03-21T13:11:54.997858+0000","last_clean":"2024-03-21T13:11:54.997858+0000","last_became_active":"2024-03-21T13:11:22.319493+0000","last_became_peered":"2024-03-21T13:11:22.319493+0000","last_unstale":"2024-03-21T13:11:54.997858+0000","last_undegraded":"2024-03-21T13:11:54.997858+0000","last_fullsized":"2024-03-21T13:11:54.997858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:04:24.024296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846413+0000","last_change":"2024-03-21T13:11:20.321173+0000","last_active":"2024-03-21T13:11:54.846413+0000","last_peered":"2024-03-21T13:11:54.846413+0000","last_clean":"2024-03-21T13:11:54.846413+0000","last_became_active":"2024-03-21T13:11:20.320841+0000","last_became_peered":"2024-03-21T13:11:20.320841+0000","last_unstale":"2024-03-21T13:11:54.846413+0000","last_undegraded":"2024-03-21T13:11:54.846413+0000","last_fullsized":"2024-03-21T13:11:54.846413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:29:23.173402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846746+0000","last_change":"2024-03-21T13:11:16.145158+0000","last_active":"2024-03-21T13:11:54.846746+0000","last_peered":"2024-03-21T13:11:54.846746+0000","last_clean":"2024-03-21T13:11:54.846746+0000","last_became_active":"2024-03-21T13:11:16.144851+0000","last_became_peered":"2024-03-21T13:11:16.144851+0000","last_unstale":"2024-03-21T13:11:54.846746+0000","last_undegraded":"2024-03-21T13:11:54.846746+0000","last_fullsized":"2024-03-21T13:11:54.846746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:22:50.803508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749497+0000","last_change":"2024-03-21T13:11:18.300903+0000","last_active":"2024-03-21T13:11:54.749497+0000","last_peered":"2024-03-21T13:11:54.749497+0000","last_clean":"2024-03-21T13:11:54.749497+0000","last_became_active":"2024-03-21T13:11:18.300472+0000","last_became_peered":"2024-03-21T13:11:18.300472+0000","last_unstale":"2024-03-21T13:11:54.749497+0000","last_undegraded":"2024-03-21T13:11:54.749497+0000","last_fullsized":"2024-03-21T13:11:54.749497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:19:02.076821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846248+0000","last_change":"2024-03-21T13:11:22.327344+0000","last_active":"2024-03-21T13:11:54.846248+0000","last_peered":"2024-03-21T13:11:54.846248+0000","last_clean":"2024-03-21T13:11:54.846248+0000","last_became_active":"2024-03-21T13:11:22.327000+0000","last_became_peered":"2024-03-21T13:11:22.327000+0000","last_unstale":"2024-03-21T13:11:54.846248+0000","last_undegraded":"2024-03-21T13:11:54.846248+0000","last_fullsized":"2024-03-21T13:11:54.846248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:18:33.539629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856244+0000","last_change":"2024-03-21T13:11:20.316689+0000","last_active":"2024-03-21T13:11:54.856244+0000","last_peered":"2024-03-21T13:11:54.856244+0000","last_clean":"2024-03-21T13:11:54.856244+0000","last_became_active":"2024-03-21T13:11:20.316301+0000","last_became_peered":"2024-03-21T13:11:20.316301+0000","last_unstale":"2024-03-21T13:11:54.856244+0000","last_undegraded":"2024-03-21T13:11:54.856244+0000","last_fullsized":"2024-03-21T13:11:54.856244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:44:08.398213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846704+0000","last_change":"2024-03-21T13:11:16.146296+0000","last_active":"2024-03-21T13:11:54.846704+0000","last_peered":"2024-03-21T13:11:54.846704+0000","last_clean":"2024-03-21T13:11:54.846704+0000","last_became_active":"2024-03-21T13:11:16.145782+0000","last_became_peered":"2024-03-21T13:11:16.145782+0000","last_unstale":"2024-03-21T13:11:54.846704+0000","last_undegraded":"2024-03-21T13:11:54.846704+0000","last_fullsized":"2024-03-21T13:11:54.846704+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:26:01.867415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":51,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748455+0000","last_change":"2024-03-21T13:11:44.604290+0000","last_active":"2024-03-21T13:11:54.748455+0000","last_peered":"2024-03-21T13:11:54.748455+0000","last_clean":"2024-03-21T13:11:54.748455+0000","last_became_active":"2024-03-21T13:11:44.593947+0000","last_became_peered":"2024-03-21T13:11:44.593947+0000","last_unstale":"2024-03-21T13:11:54.748455+0000","last_undegraded":"2024-03-21T13:11:54.748455+0000","last_fullsized":"2024-03-21T13:11:54.748455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:35.758885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749237+0000","last_change":"2024-03-21T13:11:22.517477+0000","last_active":"2024-03-21T13:11:54.749237+0000","last_peered":"2024-03-21T13:11:54.749237+0000","last_clean":"2024-03-21T13:11:54.749237+0000","last_became_active":"2024-03-21T13:11:22.517162+0000","last_became_peered":"2024-03-21T13:11:22.517162+0000","last_unstale":"2024-03-21T13:11:54.749237+0000","last_undegraded":"2024-03-21T13:11:54.749237+0000","last_fullsized":"2024-03-21T13:11:54.749237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:21:23.488718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999001+0000","last_change":"2024-03-21T13:11:20.311348+0000","last_active":"2024-03-21T13:11:54.999001+0000","last_peered":"2024-03-21T13:11:54.999001+0000","last_clean":"2024-03-21T13:11:54.999001+0000","last_became_active":"2024-03-21T13:11:20.310699+0000","last_became_peered":"2024-03-21T13:11:20.310699+0000","last_unstale":"2024-03-21T13:11:54.999001+0000","last_undegraded":"2024-03-21T13:11:54.999001+0000","last_fullsized":"2024-03-21T13:11:54.999001+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:28:35.331844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998326+0000","last_change":"2024-03-21T13:11:16.142611+0000","last_active":"2024-03-21T13:11:54.998326+0000","last_peered":"2024-03-21T13:11:54.998326+0000","last_clean":"2024-03-21T13:11:54.998326+0000","last_became_active":"2024-03-21T13:11:16.142299+0000","last_became_peered":"2024-03-21T13:11:16.142299+0000","last_unstale":"2024-03-21T13:11:54.998326+0000","last_undegraded":"2024-03-21T13:11:54.998326+0000","last_fullsized":"2024-03-21T13:11:54.998326+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:52:24.309637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744077+0000","last_change":"2024-03-21T13:11:18.302257+0000","last_active":"2024-03-21T13:11:54.744077+0000","last_peered":"2024-03-21T13:11:54.744077+0000","last_clean":"2024-03-21T13:11:54.744077+0000","last_became_active":"2024-03-21T13:11:18.301913+0000","last_became_peered":"2024-03-21T13:11:18.301913+0000","last_unstale":"2024-03-21T13:11:54.744077+0000","last_undegraded":"2024-03-21T13:11:54.744077+0000","last_fullsized":"2024-03-21T13:11:54.744077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:59:05.674276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748903+0000","last_change":"2024-03-21T13:11:22.517065+0000","last_active":"2024-03-21T13:11:54.748903+0000","last_peered":"2024-03-21T13:11:54.748903+0000","last_clean":"2024-03-21T13:11:54.748903+0000","last_became_active":"2024-03-21T13:11:22.516450+0000","last_became_peered":"2024-03-21T13:11:22.516450+0000","last_unstale":"2024-03-21T13:11:54.748903+0000","last_undegraded":"2024-03-21T13:11:54.748903+0000","last_fullsized":"2024-03-21T13:11:54.748903+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:51:12.591461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748305+0000","last_change":"2024-03-21T13:11:20.314859+0000","last_active":"2024-03-21T13:11:54.748305+0000","last_peered":"2024-03-21T13:11:54.748305+0000","last_clean":"2024-03-21T13:11:54.748305+0000","last_became_active":"2024-03-21T13:11:20.314529+0000","last_became_peered":"2024-03-21T13:11:20.314529+0000","last_unstale":"2024-03-21T13:11:54.748305+0000","last_undegraded":"2024-03-21T13:11:54.748305+0000","last_fullsized":"2024-03-21T13:11:54.748305+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:19:20.260436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749612+0000","last_change":"2024-03-21T13:11:16.145821+0000","last_active":"2024-03-21T13:11:54.749612+0000","last_peered":"2024-03-21T13:11:54.749612+0000","last_clean":"2024-03-21T13:11:54.749612+0000","last_became_active":"2024-03-21T13:11:16.145518+0000","last_became_peered":"2024-03-21T13:11:16.145518+0000","last_unstale":"2024-03-21T13:11:54.749612+0000","last_undegraded":"2024-03-21T13:11:54.749612+0000","last_fullsized":"2024-03-21T13:11:54.749612+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:28:34.069607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":52,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.747627+0000","last_change":"2024-03-21T13:11:18.301984+0000","last_active":"2024-03-21T13:11:54.747627+0000","last_peered":"2024-03-21T13:11:54.747627+0000","last_clean":"2024-03-21T13:11:54.747627+0000","last_became_active":"2024-03-21T13:11:18.301672+0000","last_became_peered":"2024-03-21T13:11:18.301672+0000","last_unstale":"2024-03-21T13:11:54.747627+0000","last_undegraded":"2024-03-21T13:11:54.747627+0000","last_fullsized":"2024-03-21T13:11:54.747627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:46:01.292003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856487+0000","last_change":"2024-03-21T13:11:22.325445+0000","last_active":"2024-03-21T13:11:54.856487+0000","last_peered":"2024-03-21T13:11:54.856487+0000","last_clean":"2024-03-21T13:11:54.856487+0000","last_became_active":"2024-03-21T13:11:22.324838+0000","last_became_peered":"2024-03-21T13:11:22.324838+0000","last_unstale":"2024-03-21T13:11:54.856487+0000","last_undegraded":"2024-03-21T13:11:54.856487+0000","last_fullsized":"2024-03-21T13:11:54.856487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:09:10.134796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210842+0000","last_change":"2024-03-21T13:11:20.312952+0000","last_active":"2024-03-21T13:12:31.210842+0000","last_peered":"2024-03-21T13:12:31.210842+0000","last_clean":"2024-03-21T13:12:31.210842+0000","last_became_active":"2024-03-21T13:11:20.312083+0000","last_became_peered":"2024-03-21T13:11:20.312083+0000","last_unstale":"2024-03-21T13:12:31.210842+0000","last_undegraded":"2024-03-21T13:12:31.210842+0000","last_fullsized":"2024-03-21T13:12:31.210842+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:55:31.135729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998065+0000","last_change":"2024-03-21T13:11:16.150340+0000","last_active":"2024-03-21T13:11:54.998065+0000","last_peered":"2024-03-21T13:11:54.998065+0000","last_clean":"2024-03-21T13:11:54.998065+0000","last_became_active":"2024-03-21T13:11:16.150025+0000","last_became_peered":"2024-03-21T13:11:16.150025+0000","last_unstale":"2024-03-21T13:11:54.998065+0000","last_undegraded":"2024-03-21T13:11:54.998065+0000","last_fullsized":"2024-03-21T13:11:54.998065+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:53:34.844020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998194+0000","last_change":"2024-03-21T13:11:18.517918+0000","last_active":"2024-03-21T13:11:54.998194+0000","last_peered":"2024-03-21T13:11:54.998194+0000","last_clean":"2024-03-21T13:11:54.998194+0000","last_became_active":"2024-03-21T13:11:18.517378+0000","last_became_peered":"2024-03-21T13:11:18.517378+0000","last_unstale":"2024-03-21T13:11:54.998194+0000","last_undegraded":"2024-03-21T13:11:54.998194+0000","last_fullsized":"2024-03-21T13:11:54.998194+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:46:00.357781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999112+0000","last_change":"2024-03-21T13:11:22.319805+0000","last_active":"2024-03-21T13:11:54.999112+0000","last_peered":"2024-03-21T13:11:54.999112+0000","last_clean":"2024-03-21T13:11:54.999112+0000","last_became_active":"2024-03-21T13:11:22.319425+0000","last_became_peered":"2024-03-21T13:11:22.319425+0000","last_unstale":"2024-03-21T13:11:54.999112+0000","last_undegraded":"2024-03-21T13:11:54.999112+0000","last_fullsized":"2024-03-21T13:11:54.999112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:27:33.427424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998758+0000","last_change":"2024-03-21T13:11:20.315555+0000","last_active":"2024-03-21T13:11:54.998758+0000","last_peered":"2024-03-21T13:11:54.998758+0000","last_clean":"2024-03-21T13:11:54.998758+0000","last_became_active":"2024-03-21T13:11:20.315111+0000","last_became_peered":"2024-03-21T13:11:20.315111+0000","last_unstale":"2024-03-21T13:11:54.998758+0000","last_undegraded":"2024-03-21T13:11:54.998758+0000","last_fullsized":"2024-03-21T13:11:54.998758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:55:32.803240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746041+0000","last_change":"2024-03-21T13:11:16.152332+0000","last_active":"2024-03-21T13:11:54.746041+0000","last_peered":"2024-03-21T13:11:54.746041+0000","last_clean":"2024-03-21T13:11:54.746041+0000","last_became_active":"2024-03-21T13:11:16.152009+0000","last_became_peered":"2024-03-21T13:11:16.152009+0000","last_unstale":"2024-03-21T13:11:54.746041+0000","last_undegraded":"2024-03-21T13:11:54.746041+0000","last_fullsized":"2024-03-21T13:11:54.746041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:36:46.651590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998340+0000","last_change":"2024-03-21T13:11:18.300099+0000","last_active":"2024-03-21T13:11:54.998340+0000","last_peered":"2024-03-21T13:11:54.998340+0000","last_clean":"2024-03-21T13:11:54.998340+0000","last_became_active":"2024-03-21T13:11:18.299744+0000","last_became_peered":"2024-03-21T13:11:18.299744+0000","last_unstale":"2024-03-21T13:11:54.998340+0000","last_undegraded":"2024-03-21T13:11:54.998340+0000","last_fullsized":"2024-03-21T13:11:54.998340+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:56:38.686906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856238+0000","last_change":"2024-03-21T13:11:22.323899+0000","last_active":"2024-03-21T13:11:54.856238+0000","last_peered":"2024-03-21T13:11:54.856238+0000","last_clean":"2024-03-21T13:11:54.856238+0000","last_became_active":"2024-03-21T13:11:22.323415+0000","last_became_peered":"2024-03-21T13:11:22.323415+0000","last_unstale":"2024-03-21T13:11:54.856238+0000","last_undegraded":"2024-03-21T13:11:54.856238+0000","last_fullsized":"2024-03-21T13:11:54.856238+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:27:57.543398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997604+0000","last_change":"2024-03-21T13:11:20.318426+0000","last_active":"2024-03-21T13:11:54.997604+0000","last_peered":"2024-03-21T13:11:54.997604+0000","last_clean":"2024-03-21T13:11:54.997604+0000","last_became_active":"2024-03-21T13:11:20.318028+0000","last_became_peered":"2024-03-21T13:11:20.318028+0000","last_unstale":"2024-03-21T13:11:54.997604+0000","last_undegraded":"2024-03-21T13:11:54.997604+0000","last_fullsized":"2024-03-21T13:11:54.997604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:37:41.487058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998764+0000","last_change":"2024-03-21T13:11:16.154151+0000","last_active":"2024-03-21T13:11:54.998764+0000","last_peered":"2024-03-21T13:11:54.998764+0000","last_clean":"2024-03-21T13:11:54.998764+0000","last_became_active":"2024-03-21T13:11:16.153801+0000","last_became_peered":"2024-03-21T13:11:16.153801+0000","last_unstale":"2024-03-21T13:11:54.998764+0000","last_undegraded":"2024-03-21T13:11:54.998764+0000","last_fullsized":"2024-03-21T13:11:54.998764+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:31:39.087490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":35,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746148+0000","last_change":"2024-03-21T13:11:18.300432+0000","last_active":"2024-03-21T13:11:54.746148+0000","last_peered":"2024-03-21T13:11:54.746148+0000","last_clean":"2024-03-21T13:11:54.746148+0000","last_became_active":"2024-03-21T13:11:18.300027+0000","last_became_peered":"2024-03-21T13:11:18.300027+0000","last_unstale":"2024-03-21T13:11:54.746148+0000","last_undegraded":"2024-03-21T13:11:54.746148+0000","last_fullsized":"2024-03-21T13:11:54.746148+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:21:09.035014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749009+0000","last_change":"2024-03-21T13:11:22.517547+0000","last_active":"2024-03-21T13:11:54.749009+0000","last_peered":"2024-03-21T13:11:54.749009+0000","last_clean":"2024-03-21T13:11:54.749009+0000","last_became_active":"2024-03-21T13:11:22.517064+0000","last_became_peered":"2024-03-21T13:11:22.517064+0000","last_unstale":"2024-03-21T13:11:54.749009+0000","last_undegraded":"2024-03-21T13:11:54.749009+0000","last_fullsized":"2024-03-21T13:11:54.749009+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:11:23.292658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856450+0000","last_change":"2024-03-21T13:11:20.309847+0000","last_active":"2024-03-21T13:11:54.856450+0000","last_peered":"2024-03-21T13:11:54.856450+0000","last_clean":"2024-03-21T13:11:54.856450+0000","last_became_active":"2024-03-21T13:11:20.309263+0000","last_became_peered":"2024-03-21T13:11:20.309263+0000","last_unstale":"2024-03-21T13:11:54.856450+0000","last_undegraded":"2024-03-21T13:11:54.856450+0000","last_fullsized":"2024-03-21T13:11:54.856450+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:25.930000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846651+0000","last_change":"2024-03-21T13:11:16.145714+0000","last_active":"2024-03-21T13:11:54.846651+0000","last_peered":"2024-03-21T13:11:54.846651+0000","last_clean":"2024-03-21T13:11:54.846651+0000","last_became_active":"2024-03-21T13:11:16.145431+0000","last_became_peered":"2024-03-21T13:11:16.145431+0000","last_unstale":"2024-03-21T13:11:54.846651+0000","last_undegraded":"2024-03-21T13:11:54.846651+0000","last_fullsized":"2024-03-21T13:11:54.846651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:16:26.774169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748665+0000","last_change":"2024-03-21T13:11:18.515931+0000","last_active":"2024-03-21T13:11:54.748665+0000","last_peered":"2024-03-21T13:11:54.748665+0000","last_clean":"2024-03-21T13:11:54.748665+0000","last_became_active":"2024-03-21T13:11:18.515623+0000","last_became_peered":"2024-03-21T13:11:18.515623+0000","last_unstale":"2024-03-21T13:11:54.748665+0000","last_undegraded":"2024-03-21T13:11:54.748665+0000","last_fullsized":"2024-03-21T13:11:54.748665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:12:32.702235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997497+0000","last_change":"2024-03-21T13:11:22.317379+0000","last_active":"2024-03-21T13:11:54.997497+0000","last_peered":"2024-03-21T13:11:54.997497+0000","last_clean":"2024-03-21T13:11:54.997497+0000","last_became_active":"2024-03-21T13:11:22.316810+0000","last_became_peered":"2024-03-21T13:11:22.316810+0000","last_unstale":"2024-03-21T13:11:54.997497+0000","last_undegraded":"2024-03-21T13:11:54.997497+0000","last_fullsized":"2024-03-21T13:11:54.997497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:54:20.727690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749232+0000","last_change":"2024-03-21T13:11:20.312629+0000","last_active":"2024-03-21T13:11:54.749232+0000","last_peered":"2024-03-21T13:11:54.749232+0000","last_clean":"2024-03-21T13:11:54.749232+0000","last_became_active":"2024-03-21T13:11:20.312314+0000","last_became_peered":"2024-03-21T13:11:20.312314+0000","last_unstale":"2024-03-21T13:11:54.749232+0000","last_undegraded":"2024-03-21T13:11:54.749232+0000","last_fullsized":"2024-03-21T13:11:54.749232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:38:18.330065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856604+0000","last_change":"2024-03-21T13:11:16.151204+0000","last_active":"2024-03-21T13:11:54.856604+0000","last_peered":"2024-03-21T13:11:54.856604+0000","last_clean":"2024-03-21T13:11:54.856604+0000","last_became_active":"2024-03-21T13:11:16.150898+0000","last_became_peered":"2024-03-21T13:11:16.150898+0000","last_unstale":"2024-03-21T13:11:54.856604+0000","last_undegraded":"2024-03-21T13:11:54.856604+0000","last_fullsized":"2024-03-21T13:11:54.856604+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:17:21.887091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998629+0000","last_change":"2024-03-21T13:11:18.296824+0000","last_active":"2024-03-21T13:11:54.998629+0000","last_peered":"2024-03-21T13:11:54.998629+0000","last_clean":"2024-03-21T13:11:54.998629+0000","last_became_active":"2024-03-21T13:11:18.296208+0000","last_became_peered":"2024-03-21T13:11:18.296208+0000","last_unstale":"2024-03-21T13:11:54.998629+0000","last_undegraded":"2024-03-21T13:11:54.998629+0000","last_fullsized":"2024-03-21T13:11:54.998629+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:47:30.252913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746721+0000","last_change":"2024-03-21T13:11:22.329711+0000","last_active":"2024-03-21T13:11:54.746721+0000","last_peered":"2024-03-21T13:11:54.746721+0000","last_clean":"2024-03-21T13:11:54.746721+0000","last_became_active":"2024-03-21T13:11:22.329391+0000","last_became_peered":"2024-03-21T13:11:22.329391+0000","last_unstale":"2024-03-21T13:11:54.746721+0000","last_undegraded":"2024-03-21T13:11:54.746721+0000","last_fullsized":"2024-03-21T13:11:54.746721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:44:57.425497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998540+0000","last_change":"2024-03-21T13:11:20.313852+0000","last_active":"2024-03-21T13:11:54.998540+0000","last_peered":"2024-03-21T13:11:54.998540+0000","last_clean":"2024-03-21T13:11:54.998540+0000","last_became_active":"2024-03-21T13:11:20.312718+0000","last_became_peered":"2024-03-21T13:11:20.312718+0000","last_unstale":"2024-03-21T13:11:54.998540+0000","last_undegraded":"2024-03-21T13:11:54.998540+0000","last_fullsized":"2024-03-21T13:11:54.998540+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:17:43.302267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748570+0000","last_change":"2024-03-21T13:11:16.145627+0000","last_active":"2024-03-21T13:11:54.748570+0000","last_peered":"2024-03-21T13:11:54.748570+0000","last_clean":"2024-03-21T13:11:54.748570+0000","last_became_active":"2024-03-21T13:11:16.145334+0000","last_became_peered":"2024-03-21T13:11:16.145334+0000","last_unstale":"2024-03-21T13:11:54.748570+0000","last_undegraded":"2024-03-21T13:11:54.748570+0000","last_fullsized":"2024-03-21T13:11:54.748570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:21.993340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748709+0000","last_change":"2024-03-21T13:11:18.296351+0000","last_active":"2024-03-21T13:11:54.748709+0000","last_peered":"2024-03-21T13:11:54.748709+0000","last_clean":"2024-03-21T13:11:54.748709+0000","last_became_active":"2024-03-21T13:11:18.294511+0000","last_became_peered":"2024-03-21T13:11:18.294511+0000","last_unstale":"2024-03-21T13:11:54.748709+0000","last_undegraded":"2024-03-21T13:11:54.748709+0000","last_fullsized":"2024-03-21T13:11:54.748709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:29:33.073734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997784+0000","last_change":"2024-03-21T13:11:22.326706+0000","last_active":"2024-03-21T13:11:54.997784+0000","last_peered":"2024-03-21T13:11:54.997784+0000","last_clean":"2024-03-21T13:11:54.997784+0000","last_became_active":"2024-03-21T13:11:22.326379+0000","last_became_peered":"2024-03-21T13:11:22.326379+0000","last_unstale":"2024-03-21T13:11:54.997784+0000","last_undegraded":"2024-03-21T13:11:54.997784+0000","last_fullsized":"2024-03-21T13:11:54.997784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:46:48.406961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210868+0000","last_change":"2024-03-21T13:11:20.317767+0000","last_active":"2024-03-21T13:12:31.210868+0000","last_peered":"2024-03-21T13:12:31.210868+0000","last_clean":"2024-03-21T13:12:31.210868+0000","last_became_active":"2024-03-21T13:11:20.317461+0000","last_became_peered":"2024-03-21T13:11:20.317461+0000","last_unstale":"2024-03-21T13:12:31.210868+0000","last_undegraded":"2024-03-21T13:12:31.210868+0000","last_fullsized":"2024-03-21T13:12:31.210868+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:38.172483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856348+0000","last_change":"2024-03-21T13:11:16.149368+0000","last_active":"2024-03-21T13:11:54.856348+0000","last_peered":"2024-03-21T13:11:54.856348+0000","last_clean":"2024-03-21T13:11:54.856348+0000","last_became_active":"2024-03-21T13:11:16.149083+0000","last_became_peered":"2024-03-21T13:11:16.149083+0000","last_unstale":"2024-03-21T13:11:54.856348+0000","last_undegraded":"2024-03-21T13:11:54.856348+0000","last_fullsized":"2024-03-21T13:11:54.856348+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:54:39.456177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.747157+0000","last_change":"2024-03-21T13:11:18.515660+0000","last_active":"2024-03-21T13:11:54.747157+0000","last_peered":"2024-03-21T13:11:54.747157+0000","last_clean":"2024-03-21T13:11:54.747157+0000","last_became_active":"2024-03-21T13:11:18.515363+0000","last_became_peered":"2024-03-21T13:11:18.515363+0000","last_unstale":"2024-03-21T13:11:54.747157+0000","last_undegraded":"2024-03-21T13:11:54.747157+0000","last_fullsized":"2024-03-21T13:11:54.747157+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:46:50.492693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749541+0000","last_change":"2024-03-21T13:11:22.323487+0000","last_active":"2024-03-21T13:11:54.749541+0000","last_peered":"2024-03-21T13:11:54.749541+0000","last_clean":"2024-03-21T13:11:54.749541+0000","last_became_active":"2024-03-21T13:11:22.323160+0000","last_became_peered":"2024-03-21T13:11:22.323160+0000","last_unstale":"2024-03-21T13:11:54.749541+0000","last_undegraded":"2024-03-21T13:11:54.749541+0000","last_fullsized":"2024-03-21T13:11:54.749541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:11:15.105135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210655+0000","last_change":"2024-03-21T13:11:20.311784+0000","last_active":"2024-03-21T13:12:31.210655+0000","last_peered":"2024-03-21T13:12:31.210655+0000","last_clean":"2024-03-21T13:12:31.210655+0000","last_became_active":"2024-03-21T13:11:20.311363+0000","last_became_peered":"2024-03-21T13:11:20.311363+0000","last_unstale":"2024-03-21T13:12:31.210655+0000","last_undegraded":"2024-03-21T13:12:31.210655+0000","last_fullsized":"2024-03-21T13:12:31.210655+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:55:47.547824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997934+0000","last_change":"2024-03-21T13:11:16.153646+0000","last_active":"2024-03-21T13:11:54.997934+0000","last_peered":"2024-03-21T13:11:54.997934+0000","last_clean":"2024-03-21T13:11:54.997934+0000","last_became_active":"2024-03-21T13:11:16.153323+0000","last_became_peered":"2024-03-21T13:11:16.153323+0000","last_unstale":"2024-03-21T13:11:54.997934+0000","last_undegraded":"2024-03-21T13:11:54.997934+0000","last_fullsized":"2024-03-21T13:11:54.997934+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:21:18.416022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.847028+0000","last_change":"2024-03-21T13:11:18.517861+0000","last_active":"2024-03-21T13:11:54.847028+0000","last_peered":"2024-03-21T13:11:54.847028+0000","last_clean":"2024-03-21T13:11:54.847028+0000","last_became_active":"2024-03-21T13:11:18.517501+0000","last_became_peered":"2024-03-21T13:11:18.517501+0000","last_unstale":"2024-03-21T13:11:54.847028+0000","last_undegraded":"2024-03-21T13:11:54.847028+0000","last_fullsized":"2024-03-21T13:11:54.847028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:08.783284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744442+0000","last_change":"2024-03-21T13:11:22.325223+0000","last_active":"2024-03-21T13:11:54.744442+0000","last_peered":"2024-03-21T13:11:54.744442+0000","last_clean":"2024-03-21T13:11:54.744442+0000","last_became_active":"2024-03-21T13:11:22.324908+0000","last_became_peered":"2024-03-21T13:11:22.324908+0000","last_unstale":"2024-03-21T13:11:54.744442+0000","last_undegraded":"2024-03-21T13:11:54.744442+0000","last_fullsized":"2024-03-21T13:11:54.744442+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:54:12.819611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748818+0000","last_change":"2024-03-21T13:11:20.310958+0000","last_active":"2024-03-21T13:11:54.748818+0000","last_peered":"2024-03-21T13:11:54.748818+0000","last_clean":"2024-03-21T13:11:54.748818+0000","last_became_active":"2024-03-21T13:11:20.310475+0000","last_became_peered":"2024-03-21T13:11:20.310475+0000","last_unstale":"2024-03-21T13:11:54.748818+0000","last_undegraded":"2024-03-21T13:11:54.748818+0000","last_fullsized":"2024-03-21T13:11:54.748818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:58:33.751690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856273+0000","last_change":"2024-03-21T13:11:16.146676+0000","last_active":"2024-03-21T13:11:54.856273+0000","last_peered":"2024-03-21T13:11:54.856273+0000","last_clean":"2024-03-21T13:11:54.856273+0000","last_became_active":"2024-03-21T13:11:16.146370+0000","last_became_peered":"2024-03-21T13:11:16.146370+0000","last_unstale":"2024-03-21T13:11:54.856273+0000","last_undegraded":"2024-03-21T13:11:54.856273+0000","last_fullsized":"2024-03-21T13:11:54.856273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:36.564709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998324+0000","last_change":"2024-03-21T13:11:18.517641+0000","last_active":"2024-03-21T13:11:54.998324+0000","last_peered":"2024-03-21T13:11:54.998324+0000","last_clean":"2024-03-21T13:11:54.998324+0000","last_became_active":"2024-03-21T13:11:18.517359+0000","last_became_peered":"2024-03-21T13:11:18.517359+0000","last_unstale":"2024-03-21T13:11:54.998324+0000","last_undegraded":"2024-03-21T13:11:54.998324+0000","last_fullsized":"2024-03-21T13:11:54.998324+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:31.385588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856558+0000","last_change":"2024-03-21T13:11:22.517376+0000","last_active":"2024-03-21T13:11:54.856558+0000","last_peered":"2024-03-21T13:11:54.856558+0000","last_clean":"2024-03-21T13:11:54.856558+0000","last_became_active":"2024-03-21T13:11:22.517100+0000","last_became_peered":"2024-03-21T13:11:22.517100+0000","last_unstale":"2024-03-21T13:11:54.856558+0000","last_undegraded":"2024-03-21T13:11:54.856558+0000","last_fullsized":"2024-03-21T13:11:54.856558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:55:05.359652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846628+0000","last_change":"2024-03-21T13:11:20.312140+0000","last_active":"2024-03-21T13:11:54.846628+0000","last_peered":"2024-03-21T13:11:54.846628+0000","last_clean":"2024-03-21T13:11:54.846628+0000","last_became_active":"2024-03-21T13:11:20.311830+0000","last_became_peered":"2024-03-21T13:11:20.311830+0000","last_unstale":"2024-03-21T13:11:54.846628+0000","last_undegraded":"2024-03-21T13:11:54.846628+0000","last_fullsized":"2024-03-21T13:11:54.846628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:13:16.882822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997619+0000","last_change":"2024-03-21T13:11:16.147247+0000","last_active":"2024-03-21T13:11:54.997619+0000","last_peered":"2024-03-21T13:11:54.997619+0000","last_clean":"2024-03-21T13:11:54.997619+0000","last_became_active":"2024-03-21T13:11:16.146940+0000","last_became_peered":"2024-03-21T13:11:16.146940+0000","last_unstale":"2024-03-21T13:11:54.997619+0000","last_undegraded":"2024-03-21T13:11:54.997619+0000","last_fullsized":"2024-03-21T13:11:54.997619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:29:54.978928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":38,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749116+0000","last_change":"2024-03-21T13:11:18.515801+0000","last_active":"2024-03-21T13:11:54.749116+0000","last_peered":"2024-03-21T13:11:54.749116+0000","last_clean":"2024-03-21T13:11:54.749116+0000","last_became_active":"2024-03-21T13:11:18.515372+0000","last_became_peered":"2024-03-21T13:11:18.515372+0000","last_unstale":"2024-03-21T13:11:54.749116+0000","last_undegraded":"2024-03-21T13:11:54.749116+0000","last_fullsized":"2024-03-21T13:11:54.749116+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:33:49.386828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846557+0000","last_change":"2024-03-21T13:11:22.517043+0000","last_active":"2024-03-21T13:11:54.846557+0000","last_peered":"2024-03-21T13:11:54.846557+0000","last_clean":"2024-03-21T13:11:54.846557+0000","last_became_active":"2024-03-21T13:11:22.516672+0000","last_became_peered":"2024-03-21T13:11:22.516672+0000","last_unstale":"2024-03-21T13:11:54.846557+0000","last_undegraded":"2024-03-21T13:11:54.846557+0000","last_fullsized":"2024-03-21T13:11:54.846557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:03.433181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856873+0000","last_change":"2024-03-21T13:11:16.149051+0000","last_active":"2024-03-21T13:11:54.856873+0000","last_peered":"2024-03-21T13:11:54.856873+0000","last_clean":"2024-03-21T13:11:54.856873+0000","last_became_active":"2024-03-21T13:11:16.148552+0000","last_became_peered":"2024-03-21T13:11:16.148552+0000","last_unstale":"2024-03-21T13:11:54.856873+0000","last_undegraded":"2024-03-21T13:11:54.856873+0000","last_fullsized":"2024-03-21T13:11:54.856873+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:03:14.897415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846261+0000","last_change":"2024-03-21T13:11:18.518098+0000","last_active":"2024-03-21T13:11:54.846261+0000","last_peered":"2024-03-21T13:11:54.846261+0000","last_clean":"2024-03-21T13:11:54.846261+0000","last_became_active":"2024-03-21T13:11:18.517747+0000","last_became_peered":"2024-03-21T13:11:18.517747+0000","last_unstale":"2024-03-21T13:11:54.846261+0000","last_undegraded":"2024-03-21T13:11:54.846261+0000","last_fullsized":"2024-03-21T13:11:54.846261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:11:46.628125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210841+0000","last_change":"2024-03-21T13:11:20.314260+0000","last_active":"2024-03-21T13:12:31.210841+0000","last_peered":"2024-03-21T13:12:31.210841+0000","last_clean":"2024-03-21T13:12:31.210841+0000","last_became_active":"2024-03-21T13:11:20.313944+0000","last_became_peered":"2024-03-21T13:11:20.313944+0000","last_unstale":"2024-03-21T13:12:31.210841+0000","last_undegraded":"2024-03-21T13:12:31.210841+0000","last_fullsized":"2024-03-21T13:12:31.210841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:18:26.325206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749070+0000","last_change":"2024-03-21T13:11:22.330740+0000","last_active":"2024-03-21T13:11:54.749070+0000","last_peered":"2024-03-21T13:11:54.749070+0000","last_clean":"2024-03-21T13:11:54.749070+0000","last_became_active":"2024-03-21T13:11:22.330243+0000","last_became_peered":"2024-03-21T13:11:22.330243+0000","last_unstale":"2024-03-21T13:11:54.749070+0000","last_undegraded":"2024-03-21T13:11:54.749070+0000","last_fullsized":"2024-03-21T13:11:54.749070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:08:57.495977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856837+0000","last_change":"2024-03-21T13:11:16.147817+0000","last_active":"2024-03-21T13:11:54.856837+0000","last_peered":"2024-03-21T13:11:54.856837+0000","last_clean":"2024-03-21T13:11:54.856837+0000","last_became_active":"2024-03-21T13:11:16.147266+0000","last_became_peered":"2024-03-21T13:11:16.147266+0000","last_unstale":"2024-03-21T13:11:54.856837+0000","last_undegraded":"2024-03-21T13:11:54.856837+0000","last_fullsized":"2024-03-21T13:11:54.856837+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:01:21.067524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":52,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748554+0000","last_change":"2024-03-21T13:11:18.305238+0000","last_active":"2024-03-21T13:11:54.748554+0000","last_peered":"2024-03-21T13:11:54.748554+0000","last_clean":"2024-03-21T13:11:54.748554+0000","last_became_active":"2024-03-21T13:11:18.304939+0000","last_became_peered":"2024-03-21T13:11:18.304939+0000","last_unstale":"2024-03-21T13:11:54.748554+0000","last_undegraded":"2024-03-21T13:11:54.748554+0000","last_fullsized":"2024-03-21T13:11:54.748554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:19:27.422003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.743826+0000","last_change":"2024-03-21T13:11:20.311620+0000","last_active":"2024-03-21T13:11:54.743826+0000","last_peered":"2024-03-21T13:11:54.743826+0000","last_clean":"2024-03-21T13:11:54.743826+0000","last_became_active":"2024-03-21T13:11:20.311207+0000","last_became_peered":"2024-03-21T13:11:20.311207+0000","last_unstale":"2024-03-21T13:11:54.743826+0000","last_undegraded":"2024-03-21T13:11:54.743826+0000","last_fullsized":"2024-03-21T13:11:54.743826+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:47.585866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856080+0000","last_change":"2024-03-21T13:11:22.326591+0000","last_active":"2024-03-21T13:11:54.856080+0000","last_peered":"2024-03-21T13:11:54.856080+0000","last_clean":"2024-03-21T13:11:54.856080+0000","last_became_active":"2024-03-21T13:11:22.326308+0000","last_became_peered":"2024-03-21T13:11:22.326308+0000","last_unstale":"2024-03-21T13:11:54.856080+0000","last_undegraded":"2024-03-21T13:11:54.856080+0000","last_fullsized":"2024-03-21T13:11:54.856080+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:24:24.747360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744051+0000","last_change":"2024-03-21T13:11:20.311279+0000","last_active":"2024-03-21T13:11:54.744051+0000","last_peered":"2024-03-21T13:11:54.744051+0000","last_clean":"2024-03-21T13:11:54.744051+0000","last_became_active":"2024-03-21T13:11:20.310943+0000","last_became_peered":"2024-03-21T13:11:20.310943+0000","last_unstale":"2024-03-21T13:11:54.744051+0000","last_undegraded":"2024-03-21T13:11:54.744051+0000","last_fullsized":"2024-03-21T13:11:54.744051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:19:23.005768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856762+0000","last_change":"2024-03-21T13:11:16.147924+0000","last_active":"2024-03-21T13:11:54.856762+0000","last_peered":"2024-03-21T13:11:54.856762+0000","last_clean":"2024-03-21T13:11:54.856762+0000","last_became_active":"2024-03-21T13:11:16.147620+0000","last_became_peered":"2024-03-21T13:11:16.147620+0000","last_unstale":"2024-03-21T13:11:54.856762+0000","last_undegraded":"2024-03-21T13:11:54.856762+0000","last_fullsized":"2024-03-21T13:11:54.856762+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:46.322538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997835+0000","last_change":"2024-03-21T13:11:18.516046+0000","last_active":"2024-03-21T13:11:54.997835+0000","last_peered":"2024-03-21T13:11:54.997835+0000","last_clean":"2024-03-21T13:11:54.997835+0000","last_became_active":"2024-03-21T13:11:18.515653+0000","last_became_peered":"2024-03-21T13:11:18.515653+0000","last_unstale":"2024-03-21T13:11:54.997835+0000","last_undegraded":"2024-03-21T13:11:54.997835+0000","last_fullsized":"2024-03-21T13:11:54.997835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:26:16.712573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997569+0000","last_change":"2024-03-21T13:11:22.318905+0000","last_active":"2024-03-21T13:11:54.997569+0000","last_peered":"2024-03-21T13:11:54.997569+0000","last_clean":"2024-03-21T13:11:54.997569+0000","last_became_active":"2024-03-21T13:11:22.318593+0000","last_became_peered":"2024-03-21T13:11:22.318593+0000","last_unstale":"2024-03-21T13:11:54.997569+0000","last_undegraded":"2024-03-21T13:11:54.997569+0000","last_fullsized":"2024-03-21T13:11:54.997569+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:39.216740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744594+0000","last_change":"2024-03-21T13:11:20.308775+0000","last_active":"2024-03-21T13:11:54.744594+0000","last_peered":"2024-03-21T13:11:54.744594+0000","last_clean":"2024-03-21T13:11:54.744594+0000","last_became_active":"2024-03-21T13:11:20.308439+0000","last_became_peered":"2024-03-21T13:11:20.308439+0000","last_unstale":"2024-03-21T13:11:54.744594+0000","last_undegraded":"2024-03-21T13:11:54.744594+0000","last_fullsized":"2024-03-21T13:11:54.744594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:34:38.666166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846521+0000","last_change":"2024-03-21T13:11:16.143237+0000","last_active":"2024-03-21T13:11:54.846521+0000","last_peered":"2024-03-21T13:11:54.846521+0000","last_clean":"2024-03-21T13:11:54.846521+0000","last_became_active":"2024-03-21T13:11:16.142895+0000","last_became_peered":"2024-03-21T13:11:16.142895+0000","last_unstale":"2024-03-21T13:11:54.846521+0000","last_undegraded":"2024-03-21T13:11:54.846521+0000","last_fullsized":"2024-03-21T13:11:54.846521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:01:26.896341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998485+0000","last_change":"2024-03-21T13:11:18.298211+0000","last_active":"2024-03-21T13:11:54.998485+0000","last_peered":"2024-03-21T13:11:54.998485+0000","last_clean":"2024-03-21T13:11:54.998485+0000","last_became_active":"2024-03-21T13:11:18.297832+0000","last_became_peered":"2024-03-21T13:11:18.297832+0000","last_unstale":"2024-03-21T13:11:54.998485+0000","last_undegraded":"2024-03-21T13:11:54.998485+0000","last_fullsized":"2024-03-21T13:11:54.998485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:12:01.735751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748142+0000","last_change":"2024-03-21T13:11:22.330144+0000","last_active":"2024-03-21T13:11:54.748142+0000","last_peered":"2024-03-21T13:11:54.748142+0000","last_clean":"2024-03-21T13:11:54.748142+0000","last_became_active":"2024-03-21T13:11:22.329794+0000","last_became_peered":"2024-03-21T13:11:22.329794+0000","last_unstale":"2024-03-21T13:11:54.748142+0000","last_undegraded":"2024-03-21T13:11:54.748142+0000","last_fullsized":"2024-03-21T13:11:54.748142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:59:19.513165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"66'2","reported_seq":42,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846364+0000","last_change":"2024-03-21T13:11:20.312696+0000","last_active":"2024-03-21T13:11:54.846364+0000","last_peered":"2024-03-21T13:11:54.846364+0000","last_clean":"2024-03-21T13:11:54.846364+0000","last_became_active":"2024-03-21T13:11:16.149446+0000","last_became_peered":"2024-03-21T13:11:16.149446+0000","last_unstale":"2024-03-21T13:11:54.846364+0000","last_undegraded":"2024-03-21T13:11:54.846364+0000","last_fullsized":"2024-03-21T13:11:54.846364+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:53:03.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001200118,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748295+0000","last_change":"2024-03-21T13:11:16.145737+0000","last_active":"2024-03-21T13:11:54.748295+0000","last_peered":"2024-03-21T13:11:54.748295+0000","last_clean":"2024-03-21T13:11:54.748295+0000","last_became_active":"2024-03-21T13:11:16.144759+0000","last_became_peered":"2024-03-21T13:11:16.144759+0000","last_unstale":"2024-03-21T13:11:54.748295+0000","last_undegraded":"2024-03-21T13:11:54.748295+0000","last_fullsized":"2024-03-21T13:11:54.748295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:47:37.336001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210646+0000","last_change":"2024-03-21T13:11:18.302318+0000","last_active":"2024-03-21T13:12:31.210646+0000","last_peered":"2024-03-21T13:12:31.210646+0000","last_clean":"2024-03-21T13:12:31.210646+0000","last_became_active":"2024-03-21T13:11:18.302003+0000","last_became_peered":"2024-03-21T13:11:18.302003+0000","last_unstale":"2024-03-21T13:12:31.210646+0000","last_undegraded":"2024-03-21T13:12:31.210646+0000","last_fullsized":"2024-03-21T13:12:31.210646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:34:23.355563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998345+0000","last_change":"2024-03-21T13:11:20.315475+0000","last_active":"2024-03-21T13:11:54.998345+0000","last_peered":"2024-03-21T13:11:54.998345+0000","last_clean":"2024-03-21T13:11:54.998345+0000","last_became_active":"2024-03-21T13:11:20.315107+0000","last_became_peered":"2024-03-21T13:11:20.315107+0000","last_unstale":"2024-03-21T13:11:54.998345+0000","last_undegraded":"2024-03-21T13:11:54.998345+0000","last_fullsized":"2024-03-21T13:11:54.998345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:16:27.491803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856866+0000","last_change":"2024-03-21T13:11:22.516024+0000","last_active":"2024-03-21T13:11:54.856866+0000","last_peered":"2024-03-21T13:11:54.856866+0000","last_clean":"2024-03-21T13:11:54.856866+0000","last_became_active":"2024-03-21T13:11:22.515689+0000","last_became_peered":"2024-03-21T13:11:22.515689+0000","last_unstale":"2024-03-21T13:11:54.856866+0000","last_undegraded":"2024-03-21T13:11:54.856866+0000","last_fullsized":"2024-03-21T13:11:54.856866+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:21:44.792827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998734+0000","last_change":"2024-03-21T13:11:16.151238+0000","last_active":"2024-03-21T13:11:54.998734+0000","last_peered":"2024-03-21T13:11:54.998734+0000","last_clean":"2024-03-21T13:11:54.998734+0000","last_became_active":"2024-03-21T13:11:16.150674+0000","last_became_peered":"2024-03-21T13:11:16.150674+0000","last_unstale":"2024-03-21T13:11:54.998734+0000","last_undegraded":"2024-03-21T13:11:54.998734+0000","last_fullsized":"2024-03-21T13:11:54.998734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:20.741716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.743902+0000","last_change":"2024-03-21T13:11:20.300589+0000","last_active":"2024-03-21T13:11:54.743902+0000","last_peered":"2024-03-21T13:11:54.743902+0000","last_clean":"2024-03-21T13:11:54.743902+0000","last_became_active":"2024-03-21T13:11:16.140637+0000","last_became_peered":"2024-03-21T13:11:16.140637+0000","last_unstale":"2024-03-21T13:11:54.743902+0000","last_undegraded":"2024-03-21T13:11:54.743902+0000","last_fullsized":"2024-03-21T13:11:54.743902+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:59.393367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055733400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999147+0000","last_change":"2024-03-21T13:11:18.297598+0000","last_active":"2024-03-21T13:11:54.999147+0000","last_peered":"2024-03-21T13:11:54.999147+0000","last_clean":"2024-03-21T13:11:54.999147+0000","last_became_active":"2024-03-21T13:11:18.294401+0000","last_became_peered":"2024-03-21T13:11:18.294401+0000","last_unstale":"2024-03-21T13:11:54.999147+0000","last_undegraded":"2024-03-21T13:11:54.999147+0000","last_fullsized":"2024-03-21T13:11:54.999147+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:45:16.818297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744066+0000","last_change":"2024-03-21T13:11:20.312133+0000","last_active":"2024-03-21T13:11:54.744066+0000","last_peered":"2024-03-21T13:11:54.744066+0000","last_clean":"2024-03-21T13:11:54.744066+0000","last_became_active":"2024-03-21T13:11:20.311822+0000","last_became_peered":"2024-03-21T13:11:20.311822+0000","last_unstale":"2024-03-21T13:11:54.744066+0000","last_undegraded":"2024-03-21T13:11:54.744066+0000","last_fullsized":"2024-03-21T13:11:54.744066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:18:05.076077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846685+0000","last_change":"2024-03-21T13:11:22.326599+0000","last_active":"2024-03-21T13:11:54.846685+0000","last_peered":"2024-03-21T13:11:54.846685+0000","last_clean":"2024-03-21T13:11:54.846685+0000","last_became_active":"2024-03-21T13:11:22.326313+0000","last_became_peered":"2024-03-21T13:11:22.326313+0000","last_unstale":"2024-03-21T13:11:54.846685+0000","last_undegraded":"2024-03-21T13:11:54.846685+0000","last_fullsized":"2024-03-21T13:11:54.846685+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:19:24.243393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'128","reported_seq":277,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:56.847154+0000","last_change":"2024-03-21T13:11:06.402763+0000","last_active":"2024-03-21T13:11:56.847154+0000","last_peered":"2024-03-21T13:11:56.847154+0000","last_clean":"2024-03-21T13:11:56.847154+0000","last_became_active":"2024-03-21T13:11:06.402276+0000","last_became_peered":"2024-03-21T13:11:06.402276+0000","last_unstale":"2024-03-21T13:11:56.847154+0000","last_undegraded":"2024-03-21T13:11:56.847154+0000","last_fullsized":"2024-03-21T13:11:56.847154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:09:01.992315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:09:01.992315+0000","last_clean_scrub_stamp":"2024-03-21T13:09:01.992315+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:09:10.536340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746899+0000","last_change":"2024-03-21T13:11:16.149764+0000","last_active":"2024-03-21T13:11:54.746899+0000","last_peered":"2024-03-21T13:11:54.746899+0000","last_clean":"2024-03-21T13:11:54.746899+0000","last_became_active":"2024-03-21T13:11:16.149486+0000","last_became_peered":"2024-03-21T13:11:16.149486+0000","last_unstale":"2024-03-21T13:11:54.746899+0000","last_undegraded":"2024-03-21T13:11:54.746899+0000","last_fullsized":"2024-03-21T13:11:54.746899+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:27:45.992677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":68,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210831+0000","last_change":"2024-03-21T13:11:18.516031+0000","last_active":"2024-03-21T13:12:31.210831+0000","last_peered":"2024-03-21T13:12:31.210831+0000","last_clean":"2024-03-21T13:12:31.210831+0000","last_became_active":"2024-03-21T13:11:18.515668+0000","last_became_peered":"2024-03-21T13:11:18.515668+0000","last_unstale":"2024-03-21T13:12:31.210831+0000","last_undegraded":"2024-03-21T13:12:31.210831+0000","last_fullsized":"2024-03-21T13:12:31.210831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:01:27.045136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856622+0000","last_change":"2024-03-21T13:11:20.319665+0000","last_active":"2024-03-21T13:11:54.856622+0000","last_peered":"2024-03-21T13:11:54.856622+0000","last_clean":"2024-03-21T13:11:54.856622+0000","last_became_active":"2024-03-21T13:11:20.319327+0000","last_became_peered":"2024-03-21T13:11:20.319327+0000","last_unstale":"2024-03-21T13:11:54.856622+0000","last_undegraded":"2024-03-21T13:11:54.856622+0000","last_fullsized":"2024-03-21T13:11:54.856622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:56:08.644596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998615+0000","last_change":"2024-03-21T13:11:22.326840+0000","last_active":"2024-03-21T13:11:54.998615+0000","last_peered":"2024-03-21T13:11:54.998615+0000","last_clean":"2024-03-21T13:11:54.998615+0000","last_became_active":"2024-03-21T13:11:22.326549+0000","last_became_peered":"2024-03-21T13:11:22.326549+0000","last_unstale":"2024-03-21T13:11:54.998615+0000","last_undegraded":"2024-03-21T13:11:54.998615+0000","last_fullsized":"2024-03-21T13:11:54.998615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:10:53.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997737+0000","last_change":"2024-03-21T13:11:16.153450+0000","last_active":"2024-03-21T13:11:54.997737+0000","last_peered":"2024-03-21T13:11:54.997737+0000","last_clean":"2024-03-21T13:11:54.997737+0000","last_became_active":"2024-03-21T13:11:16.153127+0000","last_became_peered":"2024-03-21T13:11:16.153127+0000","last_unstale":"2024-03-21T13:11:54.997737+0000","last_undegraded":"2024-03-21T13:11:54.997737+0000","last_fullsized":"2024-03-21T13:11:54.997737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:39:49.994223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":122,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:39.730320+0000","last_change":"2024-03-21T13:11:20.307323+0000","last_active":"2024-03-21T13:12:39.730320+0000","last_peered":"2024-03-21T13:12:39.730320+0000","last_clean":"2024-03-21T13:12:39.730320+0000","last_became_active":"2024-03-21T13:11:16.147728+0000","last_became_peered":"2024-03-21T13:11:16.147728+0000","last_unstale":"2024-03-21T13:12:39.730320+0000","last_undegraded":"2024-03-21T13:12:39.730320+0000","last_fullsized":"2024-03-21T13:12:39.730320+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:35:29.397384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046438600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":47,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997888+0000","last_change":"2024-03-21T13:11:18.295445+0000","last_active":"2024-03-21T13:11:54.997888+0000","last_peered":"2024-03-21T13:11:54.997888+0000","last_clean":"2024-03-21T13:11:54.997888+0000","last_became_active":"2024-03-21T13:11:18.295126+0000","last_became_peered":"2024-03-21T13:11:18.295126+0000","last_unstale":"2024-03-21T13:11:54.997888+0000","last_undegraded":"2024-03-21T13:11:54.997888+0000","last_fullsized":"2024-03-21T13:11:54.997888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:55.259101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846952+0000","last_change":"2024-03-21T13:11:20.319609+0000","last_active":"2024-03-21T13:11:54.846952+0000","last_peered":"2024-03-21T13:11:54.846952+0000","last_clean":"2024-03-21T13:11:54.846952+0000","last_became_active":"2024-03-21T13:11:20.319321+0000","last_became_peered":"2024-03-21T13:11:20.319321+0000","last_unstale":"2024-03-21T13:11:54.846952+0000","last_undegraded":"2024-03-21T13:11:54.846952+0000","last_fullsized":"2024-03-21T13:11:54.846952+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:51:41.656959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997939+0000","last_change":"2024-03-21T13:11:22.517524+0000","last_active":"2024-03-21T13:11:54.997939+0000","last_peered":"2024-03-21T13:11:54.997939+0000","last_clean":"2024-03-21T13:11:54.997939+0000","last_became_active":"2024-03-21T13:11:22.517207+0000","last_became_peered":"2024-03-21T13:11:22.517207+0000","last_unstale":"2024-03-21T13:11:54.997939+0000","last_undegraded":"2024-03-21T13:11:54.997939+0000","last_fullsized":"2024-03-21T13:11:54.997939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:12.083567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998836+0000","last_change":"2024-03-21T13:11:16.149446+0000","last_active":"2024-03-21T13:11:54.998836+0000","last_peered":"2024-03-21T13:11:54.998836+0000","last_clean":"2024-03-21T13:11:54.998836+0000","last_became_active":"2024-03-21T13:11:16.149156+0000","last_became_peered":"2024-03-21T13:11:16.149156+0000","last_unstale":"2024-03-21T13:11:54.998836+0000","last_undegraded":"2024-03-21T13:11:54.998836+0000","last_fullsized":"2024-03-21T13:11:54.998836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:30:23.099060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":65,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998397+0000","last_change":"2024-03-21T13:11:18.515699+0000","last_active":"2024-03-21T13:11:54.998397+0000","last_peered":"2024-03-21T13:11:54.998397+0000","last_clean":"2024-03-21T13:11:54.998397+0000","last_became_active":"2024-03-21T13:11:18.515414+0000","last_became_peered":"2024-03-21T13:11:18.515414+0000","last_unstale":"2024-03-21T13:11:54.998397+0000","last_undegraded":"2024-03-21T13:11:54.998397+0000","last_fullsized":"2024-03-21T13:11:54.998397+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:01.179571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997832+0000","last_change":"2024-03-21T13:11:20.311707+0000","last_active":"2024-03-21T13:11:54.997832+0000","last_peered":"2024-03-21T13:11:54.997832+0000","last_clean":"2024-03-21T13:11:54.997832+0000","last_became_active":"2024-03-21T13:11:20.311297+0000","last_became_peered":"2024-03-21T13:11:20.311297+0000","last_unstale":"2024-03-21T13:11:54.997832+0000","last_undegraded":"2024-03-21T13:11:54.997832+0000","last_fullsized":"2024-03-21T13:11:54.997832+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:32:40.830229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997497+0000","last_change":"2024-03-21T13:11:22.327442+0000","last_active":"2024-03-21T13:11:54.997497+0000","last_peered":"2024-03-21T13:11:54.997497+0000","last_clean":"2024-03-21T13:11:54.997497+0000","last_became_active":"2024-03-21T13:11:22.327159+0000","last_became_peered":"2024-03-21T13:11:22.327159+0000","last_unstale":"2024-03-21T13:11:54.997497+0000","last_undegraded":"2024-03-21T13:11:54.997497+0000","last_fullsized":"2024-03-21T13:11:54.997497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:51.930027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846915+0000","last_change":"2024-03-21T13:11:16.140879+0000","last_active":"2024-03-21T13:11:54.846915+0000","last_peered":"2024-03-21T13:11:54.846915+0000","last_clean":"2024-03-21T13:11:54.846915+0000","last_became_active":"2024-03-21T13:11:16.140580+0000","last_became_peered":"2024-03-21T13:11:16.140580+0000","last_unstale":"2024-03-21T13:11:54.846915+0000","last_undegraded":"2024-03-21T13:11:54.846915+0000","last_fullsized":"2024-03-21T13:11:54.846915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:22:10.317407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.743875+0000","last_change":"2024-03-21T13:11:44.606939+0000","last_active":"2024-03-21T13:11:54.743875+0000","last_peered":"2024-03-21T13:11:54.743875+0000","last_clean":"2024-03-21T13:11:54.743875+0000","last_became_active":"2024-03-21T13:11:44.593687+0000","last_became_peered":"2024-03-21T13:11:44.593687+0000","last_unstale":"2024-03-21T13:11:54.743875+0000","last_undegraded":"2024-03-21T13:11:54.743875+0000","last_fullsized":"2024-03-21T13:11:54.743875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:13:04.981768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210645+0000","last_change":"2024-03-21T13:11:20.316976+0000","last_active":"2024-03-21T13:12:31.210645+0000","last_peered":"2024-03-21T13:12:31.210645+0000","last_clean":"2024-03-21T13:12:31.210645+0000","last_became_active":"2024-03-21T13:11:20.316676+0000","last_became_peered":"2024-03-21T13:11:20.316676+0000","last_unstale":"2024-03-21T13:12:31.210645+0000","last_undegraded":"2024-03-21T13:12:31.210645+0000","last_fullsized":"2024-03-21T13:12:31.210645+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:59:39.772223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.856631+0000","last_change":"2024-03-21T13:11:22.518842+0000","last_active":"2024-03-21T13:11:54.856631+0000","last_peered":"2024-03-21T13:11:54.856631+0000","last_clean":"2024-03-21T13:11:54.856631+0000","last_became_active":"2024-03-21T13:11:22.518396+0000","last_became_peered":"2024-03-21T13:11:22.518396+0000","last_unstale":"2024-03-21T13:11:54.856631+0000","last_undegraded":"2024-03-21T13:11:54.856631+0000","last_fullsized":"2024-03-21T13:11:54.856631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:24:37.971321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.998078+0000","last_change":"2024-03-21T13:11:16.147549+0000","last_active":"2024-03-21T13:11:54.998078+0000","last_peered":"2024-03-21T13:11:54.998078+0000","last_clean":"2024-03-21T13:11:54.998078+0000","last_became_active":"2024-03-21T13:11:16.147232+0000","last_became_peered":"2024-03-21T13:11:16.147232+0000","last_unstale":"2024-03-21T13:11:54.998078+0000","last_undegraded":"2024-03-21T13:11:54.998078+0000","last_fullsized":"2024-03-21T13:11:54.998078+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:11.121576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748422+0000","last_change":"2024-03-21T13:11:18.296211+0000","last_active":"2024-03-21T13:11:54.748422+0000","last_peered":"2024-03-21T13:11:54.748422+0000","last_clean":"2024-03-21T13:11:54.748422+0000","last_became_active":"2024-03-21T13:11:18.295886+0000","last_became_peered":"2024-03-21T13:11:18.295886+0000","last_unstale":"2024-03-21T13:11:54.748422+0000","last_undegraded":"2024-03-21T13:11:54.748422+0000","last_fullsized":"2024-03-21T13:11:54.748422+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:50:18.908779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210657+0000","last_change":"2024-03-21T13:11:20.307730+0000","last_active":"2024-03-21T13:12:31.210657+0000","last_peered":"2024-03-21T13:12:31.210657+0000","last_clean":"2024-03-21T13:12:31.210657+0000","last_became_active":"2024-03-21T13:11:20.307389+0000","last_became_peered":"2024-03-21T13:11:20.307389+0000","last_unstale":"2024-03-21T13:12:31.210657+0000","last_undegraded":"2024-03-21T13:12:31.210657+0000","last_fullsized":"2024-03-21T13:12:31.210657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:49:05.193349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749357+0000","last_change":"2024-03-21T13:11:22.327360+0000","last_active":"2024-03-21T13:11:54.749357+0000","last_peered":"2024-03-21T13:11:54.749357+0000","last_clean":"2024-03-21T13:11:54.749357+0000","last_became_active":"2024-03-21T13:11:22.326719+0000","last_became_peered":"2024-03-21T13:11:22.326719+0000","last_unstale":"2024-03-21T13:11:54.749357+0000","last_undegraded":"2024-03-21T13:11:54.749357+0000","last_fullsized":"2024-03-21T13:11:54.749357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:40:30.105040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748618+0000","last_change":"2024-03-21T13:11:16.152317+0000","last_active":"2024-03-21T13:11:54.748618+0000","last_peered":"2024-03-21T13:11:54.748618+0000","last_clean":"2024-03-21T13:11:54.748618+0000","last_became_active":"2024-03-21T13:11:16.152039+0000","last_became_peered":"2024-03-21T13:11:16.152039+0000","last_unstale":"2024-03-21T13:11:54.748618+0000","last_undegraded":"2024-03-21T13:11:54.748618+0000","last_fullsized":"2024-03-21T13:11:54.748618+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:09:33.839419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":61,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746349+0000","last_change":"2024-03-21T13:11:18.515974+0000","last_active":"2024-03-21T13:11:54.746349+0000","last_peered":"2024-03-21T13:11:54.746349+0000","last_clean":"2024-03-21T13:11:54.746349+0000","last_became_active":"2024-03-21T13:11:18.515526+0000","last_became_peered":"2024-03-21T13:11:18.515526+0000","last_unstale":"2024-03-21T13:11:54.746349+0000","last_undegraded":"2024-03-21T13:11:54.746349+0000","last_fullsized":"2024-03-21T13:11:54.746349+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:23:55.127766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748108+0000","last_change":"2024-03-21T13:11:20.311803+0000","last_active":"2024-03-21T13:11:54.748108+0000","last_peered":"2024-03-21T13:11:54.748108+0000","last_clean":"2024-03-21T13:11:54.748108+0000","last_became_active":"2024-03-21T13:11:20.311467+0000","last_became_peered":"2024-03-21T13:11:20.311467+0000","last_unstale":"2024-03-21T13:11:54.748108+0000","last_undegraded":"2024-03-21T13:11:54.748108+0000","last_fullsized":"2024-03-21T13:11:54.748108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:41.260619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846824+0000","last_change":"2024-03-21T13:11:22.319485+0000","last_active":"2024-03-21T13:11:54.846824+0000","last_peered":"2024-03-21T13:11:54.846824+0000","last_clean":"2024-03-21T13:11:54.846824+0000","last_became_active":"2024-03-21T13:11:22.319039+0000","last_became_peered":"2024-03-21T13:11:22.319039+0000","last_unstale":"2024-03-21T13:11:54.846824+0000","last_undegraded":"2024-03-21T13:11:54.846824+0000","last_fullsized":"2024-03-21T13:11:54.846824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:22:08.693850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.211003+0000","last_change":"2024-03-21T13:11:20.314087+0000","last_active":"2024-03-21T13:12:31.211003+0000","last_peered":"2024-03-21T13:12:31.211003+0000","last_clean":"2024-03-21T13:12:31.211003+0000","last_became_active":"2024-03-21T13:11:20.313552+0000","last_became_peered":"2024-03-21T13:11:20.313552+0000","last_unstale":"2024-03-21T13:12:31.211003+0000","last_undegraded":"2024-03-21T13:12:31.211003+0000","last_fullsized":"2024-03-21T13:12:31.211003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:23:54.518787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.746263+0000","last_change":"2024-03-21T13:11:16.148240+0000","last_active":"2024-03-21T13:11:54.746263+0000","last_peered":"2024-03-21T13:11:54.746263+0000","last_clean":"2024-03-21T13:11:54.746263+0000","last_became_active":"2024-03-21T13:11:16.147954+0000","last_became_peered":"2024-03-21T13:11:16.147954+0000","last_unstale":"2024-03-21T13:11:54.746263+0000","last_undegraded":"2024-03-21T13:11:54.746263+0000","last_fullsized":"2024-03-21T13:11:54.746263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:12:27.646021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999242+0000","last_change":"2024-03-21T13:11:18.302393+0000","last_active":"2024-03-21T13:11:54.999242+0000","last_peered":"2024-03-21T13:11:54.999242+0000","last_clean":"2024-03-21T13:11:54.999242+0000","last_became_active":"2024-03-21T13:11:18.302060+0000","last_became_peered":"2024-03-21T13:11:18.302060+0000","last_unstale":"2024-03-21T13:11:54.999242+0000","last_undegraded":"2024-03-21T13:11:54.999242+0000","last_fullsized":"2024-03-21T13:11:54.999242+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T00:31:26.456952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.747072+0000","last_change":"2024-03-21T13:11:22.516838+0000","last_active":"2024-03-21T13:11:54.747072+0000","last_peered":"2024-03-21T13:11:54.747072+0000","last_clean":"2024-03-21T13:11:54.747072+0000","last_became_active":"2024-03-21T13:11:22.516417+0000","last_became_peered":"2024-03-21T13:11:22.516417+0000","last_unstale":"2024-03-21T13:11:54.747072+0000","last_undegraded":"2024-03-21T13:11:54.747072+0000","last_fullsized":"2024-03-21T13:11:54.747072+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T23:47:12.712391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.846897+0000","last_change":"2024-03-21T13:11:20.318949+0000","last_active":"2024-03-21T13:11:54.846897+0000","last_peered":"2024-03-21T13:11:54.846897+0000","last_clean":"2024-03-21T13:11:54.846897+0000","last_became_active":"2024-03-21T13:11:20.318618+0000","last_became_peered":"2024-03-21T13:11:20.318618+0000","last_unstale":"2024-03-21T13:11:54.846897+0000","last_undegraded":"2024-03-21T13:11:54.846897+0000","last_fullsized":"2024-03-21T13:11:54.846897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:36:11.784874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748148+0000","last_change":"2024-03-21T13:11:16.143852+0000","last_active":"2024-03-21T13:11:54.748148+0000","last_peered":"2024-03-21T13:11:54.748148+0000","last_clean":"2024-03-21T13:11:54.748148+0000","last_became_active":"2024-03-21T13:11:16.143563+0000","last_became_peered":"2024-03-21T13:11:16.143563+0000","last_unstale":"2024-03-21T13:11:54.748148+0000","last_undegraded":"2024-03-21T13:11:54.748148+0000","last_fullsized":"2024-03-21T13:11:54.748148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:39:24.221692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":48,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748970+0000","last_change":"2024-03-21T13:11:18.295269+0000","last_active":"2024-03-21T13:11:54.748970+0000","last_peered":"2024-03-21T13:11:54.748970+0000","last_clean":"2024-03-21T13:11:54.748970+0000","last_became_active":"2024-03-21T13:11:18.294939+0000","last_became_peered":"2024-03-21T13:11:18.294939+0000","last_unstale":"2024-03-21T13:11:54.748970+0000","last_undegraded":"2024-03-21T13:11:54.748970+0000","last_fullsized":"2024-03-21T13:11:54.748970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:23:44.006873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.744375+0000","last_change":"2024-03-21T13:11:22.326202+0000","last_active":"2024-03-21T13:11:54.744375+0000","last_peered":"2024-03-21T13:11:54.744375+0000","last_clean":"2024-03-21T13:11:54.744375+0000","last_became_active":"2024-03-21T13:11:22.325880+0000","last_became_peered":"2024-03-21T13:11:22.325880+0000","last_unstale":"2024-03-21T13:11:54.744375+0000","last_undegraded":"2024-03-21T13:11:54.744375+0000","last_fullsized":"2024-03-21T13:11:54.744375+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:44:37.952019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.999111+0000","last_change":"2024-03-21T13:11:20.311104+0000","last_active":"2024-03-21T13:11:54.999111+0000","last_peered":"2024-03-21T13:11:54.999111+0000","last_clean":"2024-03-21T13:11:54.999111+0000","last_became_active":"2024-03-21T13:11:20.310781+0000","last_became_peered":"2024-03-21T13:11:20.310781+0000","last_unstale":"2024-03-21T13:11:54.999111+0000","last_undegraded":"2024-03-21T13:11:54.999111+0000","last_fullsized":"2024-03-21T13:11:54.999111+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:20:42.727931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.997693+0000","last_change":"2024-03-21T13:11:16.149568+0000","last_active":"2024-03-21T13:11:54.997693+0000","last_peered":"2024-03-21T13:11:54.997693+0000","last_clean":"2024-03-21T13:11:54.997693+0000","last_became_active":"2024-03-21T13:11:16.149289+0000","last_became_peered":"2024-03-21T13:11:16.149289+0000","last_unstale":"2024-03-21T13:11:54.997693+0000","last_undegraded":"2024-03-21T13:11:54.997693+0000","last_fullsized":"2024-03-21T13:11:54.997693+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:50:12.149327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":81,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749036+0000","last_change":"2024-03-21T13:11:44.601154+0000","last_active":"2024-03-21T13:11:54.749036+0000","last_peered":"2024-03-21T13:11:54.749036+0000","last_clean":"2024-03-21T13:11:54.749036+0000","last_became_active":"2024-03-21T13:11:44.592315+0000","last_became_peered":"2024-03-21T13:11:44.592315+0000","last_unstale":"2024-03-21T13:11:54.749036+0000","last_undegraded":"2024-03-21T13:11:54.749036+0000","last_fullsized":"2024-03-21T13:11:54.749036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:12:31.323995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749452+0000","last_change":"2024-03-21T13:11:22.325527+0000","last_active":"2024-03-21T13:11:54.749452+0000","last_peered":"2024-03-21T13:11:54.749452+0000","last_clean":"2024-03-21T13:11:54.749452+0000","last_became_active":"2024-03-21T13:11:22.325174+0000","last_became_peered":"2024-03-21T13:11:22.325174+0000","last_unstale":"2024-03-21T13:11:54.749452+0000","last_undegraded":"2024-03-21T13:11:54.749452+0000","last_fullsized":"2024-03-21T13:11:54.749452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:21.298367+0000","last_clean_scrub_stamp":"2024-03-21T13:11:21.298367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:20:43.595688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:12:31.210657+0000","last_change":"2024-03-21T13:11:20.314517+0000","last_active":"2024-03-21T13:12:31.210657+0000","last_peered":"2024-03-21T13:12:31.210657+0000","last_clean":"2024-03-21T13:12:31.210657+0000","last_became_active":"2024-03-21T13:11:20.314190+0000","last_became_peered":"2024-03-21T13:11:20.314190+0000","last_unstale":"2024-03-21T13:12:31.210657+0000","last_undegraded":"2024-03-21T13:12:31.210657+0000","last_fullsized":"2024-03-21T13:12:31.210657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:19.284618+0000","last_clean_scrub_stamp":"2024-03-21T13:11:19.284618+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:57:37.488059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.749239+0000","last_change":"2024-03-21T13:11:16.153355+0000","last_active":"2024-03-21T13:11:54.749239+0000","last_peered":"2024-03-21T13:11:54.749239+0000","last_clean":"2024-03-21T13:11:54.749239+0000","last_became_active":"2024-03-21T13:11:16.152983+0000","last_became_peered":"2024-03-21T13:11:16.152983+0000","last_unstale":"2024-03-21T13:11:54.749239+0000","last_undegraded":"2024-03-21T13:11:54.749239+0000","last_fullsized":"2024-03-21T13:11:54.749239+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:15.119396+0000","last_clean_scrub_stamp":"2024-03-21T13:11:15.119396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:14:31.204784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-21T13:11:54.748395+0000","last_change":"2024-03-21T13:11:18.300886+0000","last_active":"2024-03-21T13:11:54.748395+0000","last_peered":"2024-03-21T13:11:54.748395+0000","last_clean":"2024-03-21T13:11:54.748395+0000","last_became_active":"2024-03-21T13:11:18.300491+0000","last_became_peered":"2024-03-21T13:11:18.300491+0000","last_unstale":"2024-03-21T13:11:54.748395+0000","last_undegraded":"2024-03-21T13:11:54.748395+0000","last_fullsized":"2024-03-21T13:11:54.748395+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T13:11:17.274925+0000","last_clean_scrub_stamp":"2024-03-21T13:11:17.274925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:58:19.420711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168598,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":931211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430235,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712620,"statfs":{"total":95995035648,"available":95961722880,"internally_reserved":0,"allocated":1384448,"data_stored":928812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":851968,"data_stored":340022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":930042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":339580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":827392,"data_stored":339335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":831488,"data_stored":340409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691872,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":802816,"data_stored":338893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T13:12:44.334 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T13:12:44.334 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-21T13:12:44.335 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-21T13:12:44.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 shell --fsid 9c355834-e783-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-21T13:12:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:44 smithi152 ceph-mon[34051]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:44 smithi152 ceph-mon[34051]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:44.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:44 smithi089 ceph-mon[30095]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:44.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:44 smithi089 ceph-mon[30095]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:44 smithi089 ceph-mon[39209]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:44 smithi089 ceph-mon[39209]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:12:44.576 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:12:46.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:46 smithi089 ceph-mon[39209]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:46.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:46 smithi089 ceph-mon[30095]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.424+0000 7f4ac643a700 1 -- 172.21.15.89:0/2605220617 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac0102ba0 msgr2=0x7f4ac010efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.424+0000 7f4ac643a700 1 --2- 172.21.15.89:0/2605220617 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac0102ba0 0x7f4ac010efe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800b0a0 tx=0x7f4ab800b3b0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.424+0000 7f4ac643a700 1 -- 172.21.15.89:0/2605220617 shutdown_connections 2024-03-21T13:12:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.424+0000 7f4ac643a700 1 --2- 172.21.15.89:0/2605220617 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac0102ba0 0x7f4ac010efe0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.424+0000 7f4ac643a700 1 --2- 172.21.15.89:0/2605220617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 0x7f4ac01085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.424+0000 7f4ac643a700 1 --2- 172.21.15.89:0/2605220617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ac01021e0 0x7f4ac0102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.424+0000 7f4ac643a700 1 -- 172.21.15.89:0/2605220617 >> 172.21.15.89:0/2605220617 conn(0x7f4ac00792c0 msgr2=0x7f4ac00796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 -- 172.21.15.89:0/2605220617 shutdown_connections 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 -- 172.21.15.89:0/2605220617 wait complete. 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 Processor -- start 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 -- start start 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac01021e0 0x7f4ac0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ac0102ba0 0x7f4ac00749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 0x7f4ac0074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0111910 con 0x7f4ac01081c0 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0111790 con 0x7f4ac0102ba0 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.425+0000 7f4ac643a700 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0111a90 con 0x7f4ac01021e0 2024-03-21T13:12:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 0x7f4ac0074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4abffff700 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac01021e0 0x7f4ac0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 0x7f4ac0074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36618/0 (socket says 172.21.15.89:36618) 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 -- 172.21.15.89:0/3072216512 learned_addr learned my addr 172.21.15.89:0/3072216512 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4abf7fe700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ac0102ba0 0x7f4ac00749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 -- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac01021e0 msgr2=0x7f4ac0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac01021e0 0x7f4ac0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 -- 172.21.15.89:0/3072216512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ac0102ba0 msgr2=0x7f4ac00749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ac0102ba0 0x7f4ac00749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 -- 172.21.15.89:0/3072216512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac01a6280 con 0x7f4ac01081c0 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4abf7fe700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ac0102ba0 0x7f4ac00749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac49d7700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 0x7f4ac0074f10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800b070 tx=0x7f4ab8007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4abd7fa700 1 -- 172.21.15.89:0/3072216512 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8021020 con 0x7f4ac01081c0 2024-03-21T13:12:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac01a6510 con 0x7f4ac01081c0 2024-03-21T13:12:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4abd7fa700 1 -- 172.21.15.89:0/3072216512 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4ab8004c50 con 0x7f4ac01081c0 2024-03-21T13:12:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4abd7fa700 1 -- 172.21.15.89:0/3072216512 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8005050 con 0x7f4ac01081c0 2024-03-21T13:12:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.426+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac01a69c0 con 0x7f4ac01081c0 2024-03-21T13:12:46.431 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:46 smithi152 ceph-mon[34051]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.427+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac0068e50 con 0x7f4ac01081c0 2024-03-21T13:12:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.428+0000 7f4abd7fa700 1 -- 172.21.15.89:0/3072216512 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f4ab8024020 con 0x7f4ac01081c0 2024-03-21T13:12:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.428+0000 7f4abd7fa700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4aa806e340 0x7f4aa8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:12:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.428+0000 7f4abd7fa700 1 -- 172.21.15.89:0/3072216512 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f4ab800f020 con 0x7f4ac01081c0 2024-03-21T13:12:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.428+0000 7f4abffff700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4aa806e340 0x7f4aa8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:12:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.429+0000 7f4abffff700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4aa806e340 0x7f4aa8070800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0009680 tx=0x7f4ab0009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:12:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.433+0000 7f4abd7fa700 1 -- 172.21.15.89:0/3072216512 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ab805cd30 con 0x7f4ac01081c0 2024-03-21T13:12:46.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.680+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4ac01a6ce0 con 0x7f4ac01081c0 2024-03-21T13:12:46.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.681+0000 7f4abd7fa700 1 -- 172.21.15.89:0/3072216512 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4ab8060780 con 0x7f4ac01081c0 2024-03-21T13:12:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-21T13:12:46.682 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-21T13:12:46.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4aa806e340 msgr2=0x7f4aa8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4aa806e340 0x7f4aa8070800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0009680 tx=0x7f4ab0009260 comp rx=0 tx=0).stop 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 msgr2=0x7f4ac0074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 0x7f4ac0074f10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800b070 tx=0x7f4ab8007a20 comp rx=0 tx=0).stop 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 shutdown_connections 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac01081c0 0x7f4ac0074f10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f4aa806e340 0x7f4aa8070800 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ac01021e0 0x7f4ac0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 --2- 172.21.15.89:0/3072216512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ac0102ba0 0x7f4ac00749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 >> 172.21.15.89:0/3072216512 conn(0x7f4ac00792c0 msgr2=0x7f4ac0105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 shutdown_connections 2024-03-21T13:12:46.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-21T13:12:46.683+0000 7f4ac643a700 1 -- 172.21.15.89:0/3072216512 wait complete. 2024-03-21T13:12:47.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3072216512' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T13:12:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3072216512' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T13:12:47.369 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-21T13:12:47.369 INFO:tasks.cephadm:Setup complete, yielding 2024-03-21T13:12:47.370 INFO:teuthology.run_tasks:Running task workunit... 2024-03-21T13:12:47.384 INFO:tasks.workunit:Pulling workunits from ref 960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:12:47.385 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-21T13:12:47.385 DEBUG:teuthology.orchestra.run.smithi089:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-21T13:12:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:12:47.401 INFO:teuthology.orchestra.run.smithi089.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-21T13:12:47.401 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-21T13:12:47.426 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3072216512' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T13:12:47.456 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-21T13:12:47.457 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-21T13:12:47.511 INFO:tasks.workunit:timeout=3h 2024-03-21T13:12:47.511 INFO:tasks.workunit:cleanup=True 2024-03-21T13:12:47.512 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:12:47.567 INFO:tasks.workunit.client.0.smithi089.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-21T13:12:47.721 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:12:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:48.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:12:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:12:48.072 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:47 smithi152 systemd[1]: Starting Ceph grafana.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:12:48.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:12:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[76445]: ts=2024-03-21T13:12:47.721Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002425323s 2024-03-21T13:12:48.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-mon[34051]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:48.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:48.383 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 podman[67465]: 2024-03-21 13:12:48.025340595 +0000 UTC m=+0.018360363 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-21T13:12:48.383 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 podman[67465]: 2024-03-21 13:12:48.129448848 +0000 UTC m=+0.122468616 container create fe5a4418479c76337ba415619d4cac1d212e2772062b70e020f3af53b4019eec (image=quay.io/ceph/grafana:9.4.12, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:12:48.383 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 podman[67465]: 2024-03-21 13:12:48.382863949 +0000 UTC m=+0.375883732 container init fe5a4418479c76337ba415619d4cac1d212e2772062b70e020f3af53b4019eec (image=quay.io/ceph/grafana:9.4.12, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:12:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:48 smithi089 ceph-mon[39209]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:48 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:48 smithi089 ceph-mon[30095]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:48 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 podman[67465]: 2024-03-21 13:12:48.388221698 +0000 UTC m=+0.381241480 container start fe5a4418479c76337ba415619d4cac1d212e2772062b70e020f3af53b4019eec (image=quay.io/ceph/grafana:9.4.12, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 bash[67465]: fe5a4418479c76337ba415619d4cac1d212e2772062b70e020f3af53b4019eec 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 systemd[1]: Started Ceph grafana.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540621118Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540806246Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540819023Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540822548Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-21T13:12:48.633 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540825415Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540828197Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540831319Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540833895Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540836474Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540839387Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540842338Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540844803Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540847408Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540849948Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-21T13:12:48.634 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540852534Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-21T13:12:48.635 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540855138Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-21T13:12:48.635 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540857454Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-21T13:12:48.635 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540860013Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-21T13:12:48.635 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=settings t=2024-03-21T13:12:48.540862653Z level=info msg="App mode production" 2024-03-21T13:12:48.635 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=sqlstore t=2024-03-21T13:12:48.540905457Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-21T13:12:48.635 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=sqlstore t=2024-03-21T13:12:48.540924699Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.541768875Z level=info msg="Starting DB migrations" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.541829689Z level=info msg="Executing migration" id="create migration_log table" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.542398181Z level=info msg="Executing migration" id="create user table" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.542840857Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.543303036Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.544080861Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.544854195Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.545760549Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.546551585Z level=info msg="Executing migration" id="create user table v2" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.547333463Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.547722927Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.548548447Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-21T13:12:48.636 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.549289305Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.550075947Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.552947842Z level=info msg="Executing migration" id="Update user table charset" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.555894099Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.556510763Z level=info msg="Executing migration" id="Add missing user data" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.559332674Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.559814121Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.562026293Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.564704365Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.567439164Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.570466409Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.57304496Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.573443906Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-21T13:12:48.637 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.576130722Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.578809546Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.581486101Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.584354944Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.58715603Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.587723061Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.590240034Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.592760035Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.595375276Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.595839374Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.598496265Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.601247061Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.603674182Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-21T13:12:48.638 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.604124922Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.606786027Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.60932448Z level=info msg="Executing migration" id="create star table" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.612029635Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.614507535Z level=info msg="Executing migration" id="create org table v1" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.617023469Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.619631183Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.622242138Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.624725958Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.627338342Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.630020961Z level=info msg="Executing migration" id="Update org table charset" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.630754382Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.631545275Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-21T13:12:48.639 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.632339797Z level=info msg="Executing migration" id="create dashboard table" 2024-03-21T13:12:48.640 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.63308468Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-21T13:12:48.640 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.633832546Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-21T13:12:48.640 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.634549334Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.635345283Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.636136583Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.636925008Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.638199639Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.638977803Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.639758463Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.640538483Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.641430376Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.642085116Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-21T13:12:48.885 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.642861048Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.643580776Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.644390357Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.645272204Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.645951758Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.646744325Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.647530773Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.648298201Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.649115838Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.649897093Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.65068702Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.651532196Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.652291754Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.653091597Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-21T13:12:48.886 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.653881339Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.654691048Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.65556724Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.656437385Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.657092375Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.658001312Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.660756078Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.663360296Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.666153339Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.669029246Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.671692292Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.674462698Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.677033295Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-21T13:12:48.887 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.679798398Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-21T13:12:48.888 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.682627991Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.685533051Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.688090891Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.691014447Z level=info msg="Executing migration" id="create data_source table" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.693625324Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.696493704Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.699203877Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.701910581Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-21T13:12:48.889 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.704467855Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.705690344Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.706531745Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.707415147Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.708356247Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.709096414Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.709999416Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.71083914Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.711699569Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.714516Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.715100343Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.71754653Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-21T13:12:48.890 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.720445318Z level=info msg="Executing migration" id="Add uid column" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.723240544Z level=info msg="Executing migration" id="Update uid value" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.726097103Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.728854663Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.731606681Z level=info msg="Executing migration" id="create api_key table" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.734423509Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.737402309Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.740234848Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.743094812Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.745812384Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.748714116Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.751620981Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.754485091Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-21T13:12:48.891 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.754912616Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.755335572Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.755748275Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.756183923Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.756492599Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.756820445Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.75710453Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.757710732Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.75831879Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.758617015Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.759205947Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.759794063Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.760209838Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-21T13:12:48.892 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.760554709Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.760907677Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.761344107Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.76175483Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.762164818Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.76246646Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.762716787Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.763321781Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.763951458Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.76424699Z level=info msg="Executing migration" id="create quota table v1" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.764622051Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.765024896Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.765305317Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.765694566Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-21T13:12:48.893 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.76610513Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.766696363Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.766956339Z level=info msg="Executing migration" id="create session table" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.767359952Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.767612226Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.767869611Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.768244531Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.768603946Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.768846359Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.769135096Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.769444776Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.769701411Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-21T13:12:48.894 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.770104962Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.770374934Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.770965726Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.771261636Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.771870106Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.772495062Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.772758816Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.773189605Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.773636189Z level=info msg="Executing migration" id="create alert table v1" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.776524152Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.779299541Z level=info msg="Executing migration" id="add index alert state" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.779794949Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.780224838Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-21T13:12:48.895 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.780630266Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.781182494Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.781620794Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.783377071Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.783763828Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.784293863Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.78464342Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.784988933Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.785402719Z level=info msg="Executing migration" id="Add column is_default" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.786140539Z level=info msg="Executing migration" id="Add column frequency" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.786887317Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.787627309Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.788379548Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-21T13:12:48.896 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.788797987Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.789100377Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.789391346Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.7897694Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.790236337Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.790674993Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.79354507Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.79628013Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.798994995Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.80172237Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.80445441Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.805024094Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.807815822Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.810375627Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.813005732Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-21T13:12:48.897 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.813450282Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.816279675Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.819147699Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.821850108Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.822321202Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.82512214Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.825982087Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.826805869Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.827295509Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.828212451Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.829011071Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.829853367Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-21T13:12:48.898 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.830695514Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.831651839Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.832380006Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.833254736Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.835284068Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.836131507Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.836996168Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.837906704Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.8386843Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.839469768Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.840323188Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.84110722Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.841973462Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.842878821Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-21T13:12:48.899 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.84382467Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.844637182Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.845479607Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.846325584Z level=info msg="Executing migration" id="Move region to single row" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.847169882Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.848169432Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.848883958Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.849878136Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.850645295Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.851530497Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.852468475Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.854489329Z level=info msg="Executing migration" id="create test_data table" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.856974779Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.859380784Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-21T13:12:48.900 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.862124489Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.864701982Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.867140174Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.869608Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.872039573Z level=info msg="Executing migration" id="create team table" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.874797571Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.877232369Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.879934846Z level=info msg="Executing migration" id="create team member table" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.882246255Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.884781867Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-21T13:12:48.901 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.887423083Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-21T13:12:49.135 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.890058354Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-21T13:12:49.135 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.892577635Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-21T13:12:49.135 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.895278211Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.896137303Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.896957288Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.897773136Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.898551132Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.89952093Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.900127107Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.900934674Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.901727138Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.902519992Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.903326028Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.903650208Z level=info msg="Executing migration" id="create tag table" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.904457304Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-21T13:12:49.136 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.905347719Z level=info msg="Executing migration" id="create login attempt table" 2024-03-21T13:12:49.137 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.905988099Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-21T13:12:49.137 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.90642811Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-21T13:12:49.137 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.907197912Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-21T13:12:49.137 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.909583016Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-21T13:12:49.137 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.90997431Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.910945848Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.911353802Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.912227986Z level=info msg="Executing migration" id="create user auth table" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.912640313Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.913086027Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.913990151Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.914806595Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.915669203Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.916558786Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.917467447Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.917908505Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.918787943Z level=info msg="Executing migration" id="create server_lock table" 2024-03-21T13:12:49.138 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.91959517Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.920384829Z level=info msg="Executing migration" id="create user auth token table" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.920840869Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.92163649Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.92258078Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.925408187Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.926534409Z level=info msg="Executing migration" id="create cache_data table" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.929277758Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.932154543Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.934815557Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.935330961Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.938017073Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.938323057Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-21T13:12:49.139 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.940786865Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.943525342Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.944138644Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.947023453Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.949503949Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.952117865Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.952557763Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.955246297Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.957808739Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.960588217Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.960926852Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-21T13:12:49.140 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.963570734Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.966115915Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.968721129Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.969050042Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.971801336Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.974395747Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.976912789Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.977380002Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.980015301Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.982522735Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.985026185Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.989478072Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.993729089Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.996332084Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-21T13:12:49.141 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.998797927Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-21T13:12:49.142 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:48.999768525Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-21T13:12:49.142 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.000530613Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.001416927Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.002095331Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.002540173Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.003267727Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.004270674Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.005179109Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.006076195Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.006497243Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.007325684Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.008438632Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-21T13:12:49.143 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.009367331Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.010117611Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.011029277Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.011970014Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.012872376Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.013474729Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.014375794Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.014974579Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.015716258Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.016720097Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.017688581Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.018639253Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.019680245Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.02070731Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-21T13:12:49.144 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.021002926Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.021757793Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.022725878Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.023495816Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.024545354Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.025319305Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.026328657Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.027023239Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.027479336Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.028454604Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.031262523Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.034137734Z level=info msg="Executing migration" id="create alert_image table" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.034752091Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-21T13:12:49.145 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.037587306Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.040378426Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.042943817Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.043348513Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.043416471Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.046020281Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.046386796Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.048829181Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.048970287Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.049074317Z level=info msg="alerts found to migrate" alerts=0 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.049448769Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.051640189Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.052097726Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.055022745Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-21T13:12:49.146 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.057717446Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.060687399Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.061009309Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.063966024Z level=info msg="Executing migration" id="create data_keys table" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.066616456Z level=info msg="Executing migration" id="create secrets table" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.069331524Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.074749541Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.077702476Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.080626295Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.08611766Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.091746102Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.094654966Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.097257718Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.097603773Z level=info msg="Executing migration" id="create permission table" 2024-03-21T13:12:49.147 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.099980863Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.102715861Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.105631329Z level=info msg="Executing migration" id="create role table" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.108346869Z level=info msg="Executing migration" id="add column display_name" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.109600274Z level=info msg="Executing migration" id="add column group_name" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.112294456Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.115013587Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.117547568Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.118003171Z level=info msg="Executing migration" id="create team role table" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.120805502Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.123588039Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.126265977Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-21T13:12:49.148 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.126734388Z level=info msg="Executing migration" id="create user role table" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.129517774Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.130325289Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.131054863Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.131482059Z level=info msg="Executing migration" id="create builtin role table" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.132388105Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.133207465Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.134024059Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.135197686Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.135974556Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.136772765Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-21T13:12:49.149 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.137214715Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-21T13:12:49.150 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.137652251Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-21T13:12:49.150 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.138035917Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-21T13:12:49.150 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.138485827Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-21T13:12:49.150 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.13960063Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-21T13:12:49.150 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.140026421Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-21T13:12:49.150 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.140476195Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-21T13:12:49.150 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.140755018Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.145631321Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.146192485Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.146618362Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.146945706Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.147354744Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.147784215Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.148236221Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-21T13:12:49.524 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.149408252Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.149701589Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.150126904Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.15055067Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.151027848Z level=info msg="Executing migration" id="add correlation config column" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.152147368Z level=info msg="Executing migration" id="create entity_events table" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.152578348Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.153004257Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.153082872Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.153360315Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.153432965Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.153674205Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.154112909Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.154533254Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-21T13:12:49.525 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.155352705Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.15580021Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.156643012Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.157669759Z level=info msg="Executing migration" id="Drop public config table" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.158410509Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.158833707Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.159764387Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.160703706Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.161508498Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.165271643Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.166465156Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.167654149Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.168497794Z level=info msg="Executing migration" id="add share column" 2024-03-21T13:12:49.526 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.169673148Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-21T13:12:49.527 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.170649664Z level=info msg="Executing migration" id="create file table" 2024-03-21T13:12:49.527 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.171423943Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-21T13:12:49.527 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.17225918Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-21T13:12:49.527 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.172734883Z level=info msg="Executing migration" id="create file_meta table" 2024-03-21T13:12:49.527 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.173149532Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.173974018Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.174864909Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.175291305Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.176170987Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.176789753Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.179058352Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.181957497Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.182626544Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.185420534Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-21T13:12:49.528 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.185786022Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.188197733Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.190865108Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.192023213Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.194956132Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.207913602Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.21067396Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.211155107Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.215432577Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.218162846Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.221107306Z level=info msg="Executing migration" id="create folder table" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.221524042Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.224430165Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.227177679Z level=info msg="Executing migration" id="Update folder title length" 2024-03-21T13:12:49.529 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.230018415Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-21T13:12:49.530 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=migrator t=2024-03-21T13:12:49.230593812Z level=info msg="migrations completed" performed=485 skipped=0 duration=688.781986ms 2024-03-21T13:12:49.530 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=sqlstore t=2024-03-21T13:12:49.230993945Z level=info msg="Created default organization" 2024-03-21T13:12:49.530 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=plugin.finder t=2024-03-21T13:12:49.266053881Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-21T13:12:49.530 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=secrets t=2024-03-21T13:12:49.266142481Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-21T13:12:49.530 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=query_data t=2024-03-21T13:12:49.268377479Z level=info msg="Query Service initialization" 2024-03-21T13:12:49.530 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=live.push_http t=2024-03-21T13:12:49.272455924Z level=info msg="Live Push Gateway initialization" 2024-03-21T13:12:49.600 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.600 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.600 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.601 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.601 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[30095]: Deploying daemon node-exporter.a on smithi089 2024-03-21T13:12:49.601 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.601 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.601 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.601 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.601 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:49 smithi089 ceph-mon[39209]: Deploying daemon node-exporter.a on smithi089 2024-03-21T13:12:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-mon[34051]: Deploying daemon node-exporter.a on smithi089 2024-03-21T13:12:50.067 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:49 smithi089 bash[77509]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-21T13:12:50.067 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:50 smithi089 bash[77509]: Getting image source signatures 2024-03-21T13:12:50.067 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:50 smithi089 bash[77509]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-21T13:12:50.067 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:50 smithi089 bash[77509]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-21T13:12:50.067 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:50 smithi089 bash[77509]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-21T13:12:50.321 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=infra.usagestats.collector t=2024-03-21T13:12:49.889546777Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-21T13:12:50.321 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=provisioning.datasources t=2024-03-21T13:12:49.890598361Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-21T13:12:50.321 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=provisioning.datasources t=2024-03-21T13:12:49.901720172Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=provisioning.alerting t=2024-03-21T13:12:49.912364633Z level=info msg="starting to provision alerting" 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=provisioning.alerting t=2024-03-21T13:12:49.912374822Z level=info msg="finished to provision alerting" 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=grafanaStorageLogger t=2024-03-21T13:12:49.91251554Z level=info msg="storage starting" 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=ngalert.state.manager t=2024-03-21T13:12:49.912557562Z level=info msg="Warming state cache for startup" 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=http.server t=2024-03-21T13:12:49.914207622Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=ngalert.state.manager t=2024-03-21T13:12:49.918584696Z level=info msg="State cache has been initialized" states=0 duration=6.026915ms 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=ngalert.multiorg.alertmanager t=2024-03-21T13:12:49.918615269Z level=info msg="starting MultiOrg Alertmanager" 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=ticker t=2024-03-21T13:12:49.918637645Z level=info msg=starting first_tick=2024-03-21T13:12:50Z 2024-03-21T13:12:50.322 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:12:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=sqlstore.transactions t=2024-03-21T13:12:49.951302459Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T13:12:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:50 smithi152 ceph-mon[34051]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:50 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:50 smithi089 ceph-mon[39209]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:50 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:50 smithi089 ceph-mon[30095]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:50 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:51.825 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:51 smithi089 bash[77509]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T13:12:51.825 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:51 smithi089 bash[77509]: Writing manifest to image destination 2024-03-21T13:12:52.221 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:51 smithi089 podman[77509]: 2024-03-21 13:12:51.878766438 +0000 UTC m=+2.252567446 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T13:12:52.221 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:51 smithi089 podman[77509]: 2024-03-21 13:12:51.98561179 +0000 UTC m=+2.359412779 container create b2da199a1d181e8761848852b23701c697e8f4e08cdca2266b6cbb3146ac5f06 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:12:52.221 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 podman[77509]: 2024-03-21 13:12:52.221374388 +0000 UTC m=+2.595175387 container init b2da199a1d181e8761848852b23701c697e8f4e08cdca2266b6cbb3146ac5f06 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 podman[77509]: 2024-03-21 13:12:52.227549294 +0000 UTC m=+2.601350282 container start b2da199a1d181e8761848852b23701c697e8f4e08cdca2266b6cbb3146ac5f06 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.231Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.231Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T13:12:52.573 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T13:12:52.574 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T13:12:52.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.232Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T13:12:52.576 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T13:12:52.577 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T13:12:52.578 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a[77560]: ts=2024-03-21T13:12:52.233Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T13:12:52.578 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 bash[77509]: b2da199a1d181e8761848852b23701c697e8f4e08cdca2266b6cbb3146ac5f06 2024-03-21T13:12:52.578 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:12:52 smithi089 systemd[1]: Started Ceph node-exporter.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:12:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[30095]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[39209]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:52 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.993 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:52 smithi152 ceph-mon[34051]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:52.994 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:52 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.994 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:52 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:52.994 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:52 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:53.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:53 smithi152 ceph-mon[34051]: Deploying daemon node-exporter.b on smithi152 2024-03-21T13:12:54.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:53 smithi089 ceph-mon[30095]: Deploying daemon node-exporter.b on smithi152 2024-03-21T13:12:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:53 smithi089 ceph-mon[39209]: Deploying daemon node-exporter.b on smithi152 2024-03-21T13:12:54.321 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:53 smithi152 bash[67758]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-21T13:12:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:54 smithi152 ceph-mon[34051]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:54 smithi089 ceph-mon[30095]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:54 smithi089 ceph-mon[39209]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:55.586 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:55 smithi152 bash[67758]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T13:12:55.586 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:55 smithi152 bash[67758]: Writing manifest to image destination 2024-03-21T13:12:55.932 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:55.932 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:55 smithi152 podman[67758]: 2024-03-21 13:12:55.72011754 +0000 UTC m=+2.242715189 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T13:12:55.933 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:55 smithi152 podman[67758]: 2024-03-21 13:12:55.827217695 +0000 UTC m=+2.349815340 container create 80fa3287e1972c20909b412d040ed6845b4c10751fb357d3034e62d851b72b3f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:12:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:12:56.321 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 podman[67758]: 2024-03-21 13:12:56.05456884 +0000 UTC m=+2.577166485 container init 80fa3287e1972c20909b412d040ed6845b4c10751fb357d3034e62d851b72b3f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 podman[67758]: 2024-03-21 13:12:56.060498073 +0000 UTC m=+2.583095720 container start 80fa3287e1972c20909b412d040ed6845b4c10751fb357d3034e62d851b72b3f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.064Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.064Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T13:12:56.322 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T13:12:56.323 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T13:12:56.323 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T13:12:56.323 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T13:12:56.323 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T13:12:56.323 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T13:12:56.323 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T13:12:56.323 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T13:12:56.324 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T13:12:56.325 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.065Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.066Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b[67809]: ts=2024-03-21T13:12:56.066Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 bash[67758]: 80fa3287e1972c20909b412d040ed6845b4c10751fb357d3034e62d851b72b3f 2024-03-21T13:12:56.326 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:12:56 smithi152 systemd[1]: Started Ceph node-exporter.b for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:12:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-mon[34051]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:56 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:12:57.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[30095]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[39209]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:12:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:56 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:12:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:12:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:12:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:12:58.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:12:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:12:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:58 smithi152 ceph-mon[34051]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:12:58 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:59.023 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:58 smithi089 ceph-mon[30095]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:59.023 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:12:58 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:12:59.024 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:58 smithi089 ceph-mon[39209]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:12:59.024 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:12:58 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:00 smithi152 ceph-mon[34051]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:00 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[30095]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[39209]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:00 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.262 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[39209]: Reconfiguring daemon alertmanager.a on smithi089 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.263 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:02.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:02.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T13:13:02.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:01 smithi089 ceph-mon[30095]: Reconfiguring daemon alertmanager.a on smithi089 2024-03-21T13:13:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:02.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:02.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:02.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T13:13:02.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:01 smithi152 ceph-mon[34051]: Reconfiguring daemon alertmanager.a on smithi089 2024-03-21T13:13:04.266 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:04 smithi089 ceph-mon[30095]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:04.266 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:04 smithi089 ceph-mon[39209]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:04 smithi152 ceph-mon[34051]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:05.519 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:05 smithi089 systemd[1]: Stopping Ceph alertmanager.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:13:05.801 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:05 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[76445]: ts=2024-03-21T13:13:05.519Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:13:05.801 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:05 smithi089 podman[78384]: 2024-03-21 13:13:05.539935838 +0000 UTC m=+0.124756350 container died 547cc80e49c368a0fb207d37a54965383fff0e038e9f30c75a37c1487c60daf4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:06.075 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:05 smithi089 podman[78384]: 2024-03-21 13:13:05.900752583 +0000 UTC m=+0.485573097 container cleanup 547cc80e49c368a0fb207d37a54965383fff0e038e9f30c75a37c1487c60daf4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:06.075 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:05 smithi089 bash[78384]: ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T13:13:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:06 smithi089 ceph-mon[30095]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:06 smithi089 ceph-mon[39209]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:06 smithi152 ceph-mon[34051]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:06.575 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 podman[78403]: 2024-03-21 13:13:06.160455234 +0000 UTC m=+0.619377717 container remove 547cc80e49c368a0fb207d37a54965383fff0e038e9f30c75a37c1487c60daf4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:06.575 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 podman[78403]: 2024-03-21 13:13:06.244331636 +0000 UTC m=+0.703254070 volume remove a3e1d0eb576ea14baa911ba472bae821a34c466119fcc05cbc97f7060fb7ffcc 2024-03-21T13:13:06.575 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T13:13:06.576 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 systemd[1]: Stopped Ceph alertmanager.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:13:06.576 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 systemd[1]: Starting Ceph alertmanager.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:13:06.861 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 podman[78505]: 2024-03-21 13:13:06.50044267 +0000 UTC m=+0.018557625 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-21T13:13:06.861 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 podman[78505]: 2024-03-21 13:13:06.604058788 +0000 UTC m=+0.122173745 volume create 41c3552979c1db6f21f8f70bbe21c75f467bdb928d708892a09361e8f15236c6 2024-03-21T13:13:06.861 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:06 smithi089 podman[78505]: 2024-03-21 13:13:06.679421417 +0000 UTC m=+0.197536415 container create 7c5aa7ad0cbfe98daad9faf14d53a83822789249c013b09c4fe4642f0eaec569 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:07.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 podman[78505]: 2024-03-21 13:13:07.015584631 +0000 UTC m=+0.533699595 container init 7c5aa7ad0cbfe98daad9faf14d53a83822789249c013b09c4fe4642f0eaec569 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:07.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 podman[78505]: 2024-03-21 13:13:07.021173597 +0000 UTC m=+0.539288553 container start 7c5aa7ad0cbfe98daad9faf14d53a83822789249c013b09c4fe4642f0eaec569 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:07.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.033Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-21T13:13:07.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.033Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-21T13:13:07.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.033Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.89 port=9094 2024-03-21T13:13:07.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.034Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T13:13:07.326 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.057Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T13:13:07.326 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.057Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T13:13:07.326 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.060Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-21T13:13:07.326 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:07.060Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T13:13:07.326 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 bash[78505]: 7c5aa7ad0cbfe98daad9faf14d53a83822789249c013b09c4fe4642f0eaec569 2024-03-21T13:13:07.326 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:07 smithi089 systemd[1]: Started Ceph alertmanager.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:13:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:13:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:08.000 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:13:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:13:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:08 smithi152 ceph-mon[34051]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:08 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:08 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:08 smithi152 ceph-mon[34051]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T13:13:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:08 smithi152 ceph-mon[34051]: Reconfiguring daemon prometheus.a on smithi152 2024-03-21T13:13:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:08 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[30095]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[30095]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T13:13:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[30095]: Reconfiguring daemon prometheus.a on smithi152 2024-03-21T13:13:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[39209]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[39209]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T13:13:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[39209]: Reconfiguring daemon prometheus.a on smithi152 2024-03-21T13:13:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:08 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:09.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:09 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:09.035Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000240822s 2024-03-21T13:13:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:10 smithi152 ceph-mon[34051]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:10.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:10 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:10.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:10 smithi089 ceph-mon[30095]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:10.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:10 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:10.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:10 smithi089 ceph-mon[39209]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:10.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:10 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:11.072 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:10 smithi152 systemd[1]: Stopping Ceph prometheus.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.096Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-21T13:13:11.360 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-21T13:13:11.361 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T13:13:11.361 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-21T13:13:11.361 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[65292]: ts=2024-03-21T13:13:11.097Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-21T13:13:11.361 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 podman[68846]: 2024-03-21 13:13:11.107359267 +0000 UTC m=+0.126333008 container died 77b19770aaad14118de33ecefbe1ece1f535beb863a97a78a7b9e41490e6b487 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:11.701 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 podman[68846]: 2024-03-21 13:13:11.459111261 +0000 UTC m=+0.478085001 container cleanup 77b19770aaad14118de33ecefbe1ece1f535beb863a97a78a7b9e41490e6b487 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:11.701 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 bash[68846]: ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T13:13:12.021 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 podman[68865]: 2024-03-21 13:13:11.701901027 +0000 UTC m=+0.596876606 container remove 77b19770aaad14118de33ecefbe1ece1f535beb863a97a78a7b9e41490e6b487 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:12.021 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T13:13:12.022 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 systemd[1]: Stopped Ceph prometheus.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:13:12.022 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.175s CPU time. 2024-03-21T13:13:12.022 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:11 smithi152 systemd[1]: Starting Ceph prometheus.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:13:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-mon[34051]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:12.321 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 podman[68964]: 2024-03-21 13:13:11.926616679 +0000 UTC m=+0.019689587 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-21T13:13:12.321 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 podman[68964]: 2024-03-21 13:13:12.053430502 +0000 UTC m=+0.146503383 container create 9a3530fee574c438be3a93fdd0f9b00a85c2486891957f09f261e0ab61ab11a8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:12.485 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-mon[30095]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:12.485 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-mon[39209]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:12.822 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 podman[68964]: 2024-03-21 13:13:12.321675464 +0000 UTC m=+0.414748369 container init 9a3530fee574c438be3a93fdd0f9b00a85c2486891957f09f261e0ab61ab11a8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:12.822 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 podman[68964]: 2024-03-21 13:13:12.327433928 +0000 UTC m=+0.420506812 container start 9a3530fee574c438be3a93fdd0f9b00a85c2486891957f09f261e0ab61ab11a8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:13:12.822 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.351Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-21T13:13:12.822 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.351Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-21T13:13:12.822 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.351Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi152 (none))" 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.351Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.351Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.352Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.353Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.354Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.354Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.356Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.356Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.539µs 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.356Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.357Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.357Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-21T13:13:12.823 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.357Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.194µs wal_replay_duration=610.779µs wbl_replay_duration=162ns total_replay_duration=649.128µs 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.358Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.358Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.358Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.378Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.103411ms db_storage=807ns remote_storage=1.06µs web_handler=259ns query_engine=921ns scrape=452.662µs scrape_sd=316.85µs notify=17.599µs notify_sd=10.524µs rules=18.959241ms tracing=5.797µs 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.378Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:13:12.378Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 bash[68964]: 9a3530fee574c438be3a93fdd0f9b00a85c2486891957f09f261e0ab61ab11a8 2024-03-21T13:13:12.824 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:13:12 smithi152 systemd[1]: Started Ceph prometheus.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STOPPING 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STOPPED 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STARTING 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Serving on http://:::9283 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STARTED 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STOPPING 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STOPPED 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STARTING 2024-03-21T13:13:12.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Serving on http://:::9283 2024-03-21T13:13:12.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STARTED 2024-03-21T13:13:12.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STOPPING 2024-03-21T13:13:12.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:13:12.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STOPPED 2024-03-21T13:13:12.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STARTING 2024-03-21T13:13:13.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Serving on http://:::9283 2024-03-21T13:13:13.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:12 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:13:12] ENGINE Bus STARTED 2024-03-21T13:13:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:13:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi152.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi152.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi152.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi152.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:13:13.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi152.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi152.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi152.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi152.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:13:13.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi152.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi152.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi152.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi152.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:13.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:13:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:14 smithi152 ceph-mon[34051]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:14.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:14 smithi089 ceph-mon[30095]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:14.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:14 smithi089 ceph-mon[39209]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:16 smithi152 ceph-mon[34051]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:16 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:16.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:16 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:16 smithi089 ceph-mon[30095]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:16 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:16 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:16 smithi089 ceph-mon[39209]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:13:16.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:16 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:16.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:16 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:17.325 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:13:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:13:17.036Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001329808s 2024-03-21T13:13:17.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:13:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:18.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:13:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:13:18.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:18 smithi152 ceph-mon[34051]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:18.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:18 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:18 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:18 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:18.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:18 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:18.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:18 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:18 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[30095]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:18.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[39209]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:18.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:13:18.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:13:18.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:13:18.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:18 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:20.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:20 smithi152 ceph-mon[34051]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:20.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:20 smithi089 ceph-mon[30095]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:20.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:20 smithi089 ceph-mon[39209]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:22 smithi152 ceph-mon[34051]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:22.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:22 smithi089 ceph-mon[30095]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:22.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:22 smithi089 ceph-mon[39209]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:24 smithi152 ceph-mon[34051]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:24 smithi089 ceph-mon[39209]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:24 smithi089 ceph-mon[30095]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:25.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:25 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:25.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:25 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:25.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:25 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:26.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:26 smithi152 ceph-mon[34051]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:26.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:26 smithi089 ceph-mon[30095]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:26.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:26 smithi089 ceph-mon[39209]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:13:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:28.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:13:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:13:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:28 smithi152 ceph-mon[34051]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:28 smithi089 ceph-mon[39209]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:28 smithi089 ceph-mon[30095]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:29.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:29.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:30 smithi152 ceph-mon[34051]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:30 smithi089 ceph-mon[30095]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:30 smithi089 ceph-mon[39209]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:32 smithi152 ceph-mon[34051]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:32.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:32 smithi089 ceph-mon[30095]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:32.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:32 smithi089 ceph-mon[39209]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:34 smithi152 ceph-mon[34051]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:34.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:34 smithi089 ceph-mon[30095]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:34 smithi089 ceph-mon[39209]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:36.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:36 smithi152 ceph-mon[34051]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:36 smithi089 ceph-mon[30095]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:36 smithi089 ceph-mon[39209]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:37.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:13:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:38.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:13:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:13:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:38 smithi152 ceph-mon[34051]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:38.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:38 smithi089 ceph-mon[30095]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:38.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:38 smithi089 ceph-mon[39209]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:39.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:40 smithi152 ceph-mon[34051]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:40 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:40 smithi089 ceph-mon[30095]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:40.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:40 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:40.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:40 smithi089 ceph-mon[39209]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:40.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:40 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:42 smithi152 ceph-mon[34051]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:42.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:42 smithi089 ceph-mon[39209]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:42 smithi089 ceph-mon[30095]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:44 smithi152 ceph-mon[34051]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:44 smithi089 ceph-mon[30095]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:44 smithi089 ceph-mon[39209]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:46 smithi152 ceph-mon[34051]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:46 smithi089 ceph-mon[30095]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:46 smithi089 ceph-mon[39209]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:47.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:13:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:48.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:13:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:13:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:48 smithi152 ceph-mon[34051]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:48 smithi089 ceph-mon[30095]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:48 smithi089 ceph-mon[39209]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:49.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:49.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:13:50.571 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:50 smithi089 ceph-mon[30095]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:50 smithi152 ceph-mon[34051]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:50.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:50 smithi089 ceph-mon[39209]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:52 smithi089 ceph-mon[30095]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:52 smithi089 ceph-mon[39209]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:52 smithi152 ceph-mon[34051]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:54.874 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:54 smithi152 ceph-mon[34051]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:54 smithi089 ceph-mon[30095]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:54 smithi089 ceph-mon[39209]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:56.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:13:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:56 smithi152 ceph-mon[34051]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:57.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:56 smithi089 ceph-mon[39209]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:56 smithi089 ceph-mon[30095]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:13:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:13:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:13:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:13:58.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:13:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:13:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:58 smithi152 ceph-mon[34051]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:59.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:58 smithi089 ceph-mon[30095]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:13:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:58 smithi089 ceph-mon[39209]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:13:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:00.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:13:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:00.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:13:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:00 smithi152 ceph-mon[34051]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:00 smithi089 ceph-mon[30095]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:00 smithi089 ceph-mon[39209]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:02 smithi152 ceph-mon[34051]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:02 smithi089 ceph-mon[30095]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:02 smithi089 ceph-mon[39209]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:04 smithi152 ceph-mon[34051]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:05.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:04 smithi089 ceph-mon[39209]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:04 smithi089 ceph-mon[30095]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:06 smithi152 ceph-mon[34051]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:07.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:06 smithi089 ceph-mon[30095]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:07.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:06 smithi089 ceph-mon[39209]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:14:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:14:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:08.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:14:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:14:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:08 smithi152 ceph-mon[34051]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:08 smithi089 ceph-mon[30095]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:08 smithi089 ceph-mon[39209]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:10.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:11.024 INFO:tasks.workunit.client.0.smithi089.stderr:Note: switching to '960fbbe4645c2becc8084669be194c95183c7753'. 2024-03-21T13:14:11.024 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr:state without impacting any branches by switching back to a branch. 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr: git switch -c 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-03-21T13:14:11.025 INFO:tasks.workunit.client.0.smithi089.stderr:Or undo this operation with: 2024-03-21T13:14:11.026 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-03-21T13:14:11.026 INFO:tasks.workunit.client.0.smithi089.stderr: git switch - 2024-03-21T13:14:11.026 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-03-21T13:14:11.026 INFO:tasks.workunit.client.0.smithi089.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-21T13:14:11.026 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-03-21T13:14:11.026 INFO:tasks.workunit.client.0.smithi089.stderr:HEAD is now at 960fbbe4645 Merge branch 'cv-refactor-osd-objectstore' of https://github.com/guits/ceph into wip-guits-testing-2024-03-21-0818 2024-03-21T13:14:11.029 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-21T13:14:11.067 INFO:tasks.workunit.client.0.smithi089.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-21T13:14:11.069 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-21T13:14:11.069 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-21T13:14:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:10 smithi152 ceph-mon[34051]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:10 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:11.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:10 smithi089 ceph-mon[30095]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:11.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:10 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:11.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:10 smithi089 ceph-mon[39209]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:11.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:10 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:11.859 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-21T13:14:11.918 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-21T13:14:11.959 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-21T13:14:11.961 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-21T13:14:11.961 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-21T13:14:12.001 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-21T13:14:12.003 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-21T13:14:12.003 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-21T13:14:12.059 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-21T13:14:12.060 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-21T13:14:12.061 DEBUG:teuthology.orchestra.run.smithi089:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=960fbbe4645c2becc8084669be194c95183c7753 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ parallel=1 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' '' = --serial ']' 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ crimson=0 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' '' = --crimson ']' 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ color= 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -t 1 ']' 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-21T13:14:12.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ declare -A pids 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_aio 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_aio' 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_aio 2024-03-21T13:14:12.126 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_aio 2024-03-21T13:14:12.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79355 2024-03-21T13:14:12.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_aio on pid 79355' 2024-03-21T13:14:12.127 INFO:tasks.workunit.client.0.smithi089.stdout:test api_aio on pid 79355 2024-03-21T13:14:12.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79355 2024-03-21T13:14:12.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-21T13:14:12.130 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_aio_pp 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_aio_pp' 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_aio_pp 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.131 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.132 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.133 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.133 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_aio_pp 2024-03-21T13:14:12.133 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79362 2024-03-21T13:14:12.133 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_aio_pp on pid 79362' 2024-03-21T13:14:12.133 INFO:tasks.workunit.client.0.smithi089.stdout:test api_aio_pp on pid 79362 2024-03-21T13:14:12.136 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79362 2024-03-21T13:14:12.136 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.136 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.136 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_io 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_io' 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_io 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.137 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.138 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.139 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_io 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79371 2024-03-21T13:14:12.140 INFO:tasks.workunit.client.0.smithi089.stdout:test api_io on pid 79371 2024-03-21T13:14:12.144 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_io on pid 79371' 2024-03-21T13:14:12.144 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79371 2024-03-21T13:14:12.144 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-21T13:14:12.144 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_io_pp 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_io_pp' 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_io_pp 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.145 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.146 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.147 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.148 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_io_pp 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79382 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_io_pp on pid 79382' 2024-03-21T13:14:12.149 INFO:tasks.workunit.client.0.smithi089.stdout:test api_io_pp on pid 79382 2024-03-21T13:14:12.154 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79382 2024-03-21T13:14:12.154 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.154 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-21T13:14:12.154 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_asio 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_asio' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_asio 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.155 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.156 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.157 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.158 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.159 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.160 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_asio 2024-03-21T13:14:12.161 INFO:tasks.workunit.client.0.smithi089.stdout:test api_asio on pid 79398 2024-03-21T13:14:12.168 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79398 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_asio on pid 79398' 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79398 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_list 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_list' 2024-03-21T13:14:12.169 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_list 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.170 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.171 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.172 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.173 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.174 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.175 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.176 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79355/exe 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_list 2024-03-21T13:14:12.177 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79415 2024-03-21T13:14:12.178 INFO:tasks.workunit.client.0.smithi089.stdout:test api_list on pid 79415 2024-03-21T13:14:12.183 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_list on pid 79415' 2024-03-21T13:14:12.183 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79415 2024-03-21T13:14:12.183 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.183 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.183 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-21T13:14:12.183 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_lock 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_lock' 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.184 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_lock 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.185 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.186 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.187 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.188 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.189 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_aio: /' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.190 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.191 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79362/exe 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.192 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.193 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.193 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.193 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.193 INFO:tasks.workunit.client.0.smithi089.stdout:test api_lock on pid 79435 2024-03-21T13:14:12.196 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_lock 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79435 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_lock on pid 79435' 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79435 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_lock_pp 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_lock_pp' 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.197 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.198 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.199 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.200 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.201 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_lock_pp 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.202 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.203 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.204 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.205 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.206 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_lock_pp 2024-03-21T13:14:12.206 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79371/exe 2024-03-21T13:14:12.206 INFO:tasks.workunit.client.0.smithi089.stdout:test api_lock_pp on pid 79455 2024-03-21T13:14:12.208 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79455 2024-03-21T13:14:12.208 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_lock_pp on pid 79455' 2024-03-21T13:14:12.208 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79455 2024-03-21T13:14:12.208 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.208 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.208 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_misc 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_misc' 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.209 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.210 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.211 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.212 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_io: /' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_misc 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.213 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.214 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.214 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.214 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.214 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.214 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.214 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_misc 2024-03-21T13:14:12.214 INFO:tasks.workunit.client.0.smithi089.stdout:test api_misc on pid 79468 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79468 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_misc on pid 79468' 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79468 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_misc_pp 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_misc_pp' 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79382/exe 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-21T13:14:12.220 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_io.log 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.221 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.222 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.223 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.224 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_misc_pp 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-21T13:14:12.225 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.226 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.227 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.228 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_misc_pp 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79489 2024-03-21T13:14:12.229 INFO:tasks.workunit.client.0.smithi089.stdout:test api_misc_pp on pid 79489 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_misc_pp on pid 79489' 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79489 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.233 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.234 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.235 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.236 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.237 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.238 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.239 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79415/exe 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.240 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.241 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.242 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.243 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_tier_pp 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_tier_pp' 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_list: /' 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.244 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79398/exe 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.245 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_list.log 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.246 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.247 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.248 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_tier_pp 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.249 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.250 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.250 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.250 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_asio: /' 2024-03-21T13:14:12.250 INFO:tasks.workunit.client.0.smithi089.stdout:test api_tier_pp on pid 79523 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_tier_pp 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79523 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_tier_pp on pid 79523' 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79523 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_pool 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.252 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_pool' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.253 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_pool 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.254 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.255 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.256 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.256 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.256 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.256 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_pool 2024-03-21T13:14:12.256 INFO:tasks.workunit.client.0.smithi089.stdout:test api_pool on pid 79531 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79531 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_pool on pid 79531' 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79531 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.260 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79435/exe 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.261 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.262 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.263 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.264 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.265 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_snapshots 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_snapshots' 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.266 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.267 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_lock: /' 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.268 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.269 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.270 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.271 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.272 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79468/exe 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_snapshots 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_snapshots 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.273 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.274 INFO:tasks.workunit.client.0.smithi089.stdout:test api_snapshots on pid 79577 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79577 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_snapshots on pid 79577' 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79577 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.280 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_snapshots_pp 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.281 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_snapshots_pp' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.282 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.283 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.284 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.285 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79523/exe 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.286 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_snapshots_pp 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79489/exe 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.287 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.288 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_misc: /' 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.289 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.291 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.291 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.291 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.291 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.291 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.291 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-21T13:14:12.291 INFO:tasks.workunit.client.0.smithi089.stdout:test api_snapshots_pp on pid 79655 2024-03-21T13:14:12.297 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_snapshots_pp 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79655 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_snapshots_pp on pid 79655' 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79655 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.298 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.299 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_stat 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_stat' 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.300 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.301 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.302 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.303 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.304 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.305 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.306 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.307 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.308 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.309 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_stat 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.310 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.311 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.312 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79455/exe 2024-03-21T13:14:12.313 INFO:tasks.workunit.client.0.smithi089.stdout:test api_stat on pid 79679 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_stat 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79679 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_stat on pid 79679' 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79679 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79531/exe 2024-03-21T13:14:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_stat_pp 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.317 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_stat_pp' 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_pool: /' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_stat_pp 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.319 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.320 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.322 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.323 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-21T13:14:12.325 INFO:tasks.workunit.client.0.smithi089.stdout:test api_stat_pp on pid 79713 2024-03-21T13:14:12.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_stat_pp 2024-03-21T13:14:12.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79713 2024-03-21T13:14:12.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_stat_pp on pid 79713' 2024-03-21T13:14:12.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79713 2024-03-21T13:14:12.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_watch_notify 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_watch_notify' 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79577/exe 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.332 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.333 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_watch_notify 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.335 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.336 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.338 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.338 INFO:tasks.workunit.client.0.smithi089.stdout:test api_watch_notify on pid 79752 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_watch_notify 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79752 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_watch_notify on pid 79752' 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79752 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_watch_notify_pp 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_watch_notify_pp' 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_watch_notify_pp 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.340 INFO:tasks.workunit.client.0.smithi089.stdout:test api_watch_notify_pp on pid 79765 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_watch_notify_pp 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79765 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_watch_notify_pp on pid 79765' 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79765 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.343 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.344 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_cmd 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_cmd' 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_cmd 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.346 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_cmd 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79792 2024-03-21T13:14:12.348 INFO:tasks.workunit.client.0.smithi089.stdout:test api_cmd on pid 79792 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_cmd on pid 79792' 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79792 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.353 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_cmd_pp 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_cmd_pp' 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.354 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_cmd_pp 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.356 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.358 INFO:tasks.workunit.client.0.smithi089.stdout:test api_cmd_pp on pid 79827 2024-03-21T13:14:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_cmd_pp 2024-03-21T13:14:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79827 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_cmd_pp on pid 79827' 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79827 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.366 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.367 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.369 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.370 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_service 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.372 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_service' 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79655/exe 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_service 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.374 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.375 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.376 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79679/exe 2024-03-21T13:14:12.376 INFO:tasks.workunit.client.0.smithi089.stdout:test api_service on pid 79869 2024-03-21T13:14:12.380 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_service 2024-03-21T13:14:12.380 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.380 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.380 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79869 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_service on pid 79869' 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79869 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.382 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_stat: /' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.385 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.387 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.388 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79713/exe 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.390 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_service_pp 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.391 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.393 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_service_pp' 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-21T13:14:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.395 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_service_pp 2024-03-21T13:14:12.396 INFO:tasks.workunit.client.0.smithi089.stdout:test api_service_pp on pid 79915 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_service_pp 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79915 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_service_pp on pid 79915' 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79915 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.405 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_c_write_operations 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_c_write_operations' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.406 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.408 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.410 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.411 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.412 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.413 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.414 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.415 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79752/exe 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.416 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.417 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_c_write_operations 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.418 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.419 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.419 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.419 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.419 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.419 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.419 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.420 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.420 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.420 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.420 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.421 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.422 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79827/exe 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.423 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.424 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.425 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.426 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.427 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.428 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.429 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.429 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.429 INFO:tasks.workunit.client.0.smithi089.stdout:test api_c_write_operations on pid 79967 2024-03-21T13:14:12.432 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_c_write_operations 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79967 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_c_write_operations on pid 79967' 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=79967 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.433 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79765/exe 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.434 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.435 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.436 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.437 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_c_read_operations 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_c_read_operations' 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-21T13:14:12.438 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.439 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.440 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.441 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79869/exe 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.442 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-21T13:14:12.443 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_service.log 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.444 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_service: /' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.445 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.446 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_c_read_operations 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79792/exe 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.447 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.448 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.448 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.448 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.448 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.448 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-21T13:14:12.448 INFO:tasks.workunit.client.0.smithi089.stdout:test api_c_read_operations on pid 80025 2024-03-21T13:14:12.451 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_c_read_operations 2024-03-21T13:14:12.451 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80025 2024-03-21T13:14:12.451 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_c_read_operations on pid 80025' 2024-03-21T13:14:12.451 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-21T13:14:12.451 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80025 2024-03-21T13:14:12.451 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.451 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.452 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.453 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_cmd: /' 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s api_cls_remote_reads 2024-03-21T13:14:12.454 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' api_cls_remote_reads' 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.455 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79915/exe 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.456 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.457 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo api_cls_remote_reads 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.458 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.459 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=api_cls_remote_reads 2024-03-21T13:14:12.460 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80060 2024-03-21T13:14:12.461 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test api_cls_remote_reads on pid 80060' 2024-03-21T13:14:12.461 INFO:tasks.workunit.client.0.smithi089.stdout:test api_cls_remote_reads on pid 80060 2024-03-21T13:14:12.463 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80060 2024-03-21T13:14:12.463 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.463 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.463 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/79967/exe 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s list_parallel 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.464 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' list_parallel' 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.465 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.466 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.466 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.466 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo list_parallel 2024-03-21T13:14:12.466 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.466 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.466 INFO:tasks.workunit.client.0.smithi089.stdout:test list_parallel on pid 80091 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=list_parallel 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80091 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test list_parallel on pid 80091' 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80091 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.468 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.469 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s open_pools_parallel 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' open_pools_parallel' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo open_pools_parallel 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.470 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.471 INFO:tasks.workunit.client.0.smithi089.stdout:test open_pools_parallel on pid 80111 2024-03-21T13:14:12.475 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=open_pools_parallel 2024-03-21T13:14:12.475 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80111 2024-03-21T13:14:12.475 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test open_pools_parallel on pid 80111' 2024-03-21T13:14:12.475 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80111 2024-03-21T13:14:12.475 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-21T13:14:12.475 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.475 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s delete_pools_parallel 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' delete_pools_parallel' 2024-03-21T13:14:12.476 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.477 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo delete_pools_parallel 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.478 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.479 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.480 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.481 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.482 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.483 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80025/exe 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.484 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.485 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.486 INFO:tasks.workunit.client.0.smithi089.stdout:test delete_pools_parallel on pid 80147 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=delete_pools_parallel 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.493 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80147 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test delete_pools_parallel on pid 80147' 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80147 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.494 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.495 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.496 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.497 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.498 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.499 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s cls 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' cls' 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80060/exe 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.500 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.501 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.502 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.503 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo cls 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.504 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.505 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.506 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.507 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.508 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.509 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=cls 2024-03-21T13:14:12.509 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.509 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80200 2024-03-21T13:14:12.509 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test cls on pid 80200' 2024-03-21T13:14:12.509 INFO:tasks.workunit.client.0.smithi089.stdout:test cls on pid 80200 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80200 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.512 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s cmd 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' cmd' 2024-03-21T13:14:12.513 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80111/exe 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.514 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.515 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.516 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.517 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo cmd 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.518 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=cmd 2024-03-21T13:14:12.520 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80221 2024-03-21T13:14:12.521 INFO:tasks.workunit.client.0.smithi089.stdout:test cmd on pid 80221 2024-03-21T13:14:12.524 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test cmd on pid 80221' 2024-03-21T13:14:12.524 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80221 2024-03-21T13:14:12.524 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.524 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.524 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.525 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s handler_error 2024-03-21T13:14:12.526 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' handler_error' 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80147/exe 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo handler_error 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.527 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.528 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.529 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.530 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.531 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.531 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.531 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.531 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.531 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.531 INFO:tasks.workunit.client.0.smithi089.stdout:test handler_error on pid 80263 2024-03-21T13:14:12.535 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.535 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.535 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.535 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-21T13:14:12.535 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=handler_error 2024-03-21T13:14:12.535 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80263 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test handler_error on pid 80263' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80263 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.536 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s io 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' io' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.538 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-21T13:14:12.539 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80091/exe 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.540 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.541 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo io 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.542 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ list_parallel: /' 2024-03-21T13:14:12.543 INFO:tasks.workunit.client.0.smithi089.stdout:test io on pid 80312 2024-03-21T13:14:12.544 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=io 2024-03-21T13:14:12.544 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80312 2024-03-21T13:14:12.544 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test io on pid 80312' 2024-03-21T13:14:12.544 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80312 2024-03-21T13:14:12.544 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.545 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.545 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s ec_io 2024-03-21T13:14:12.545 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' ec_io' 2024-03-21T13:14:12.545 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo ec_io 2024-03-21T13:14:12.545 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.545 INFO:tasks.workunit.client.0.smithi089.stdout:test ec_io on pid 80317 2024-03-21T13:14:12.549 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=ec_io 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80317 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test ec_io on pid 80317' 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80317 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.550 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.553 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.554 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s list 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' list' 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.555 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.556 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.558 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.559 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.560 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80221/exe 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo list 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.561 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.562 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.563 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.564 INFO:tasks.workunit.client.0.smithi089.stdout:test list on pid 80361 2024-03-21T13:14:12.568 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=list 2024-03-21T13:14:12.568 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80361 2024-03-21T13:14:12.568 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test list on pid 80361' 2024-03-21T13:14:12.568 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80361 2024-03-21T13:14:12.568 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.569 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ cmd: /' 2024-03-21T13:14:12.570 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_cmd 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.571 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s ec_list 2024-03-21T13:14:12.572 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' ec_list' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.573 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.574 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.575 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.576 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo ec_list 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.577 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.578 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.579 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.580 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.580 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.580 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.580 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.580 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.580 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.580 INFO:tasks.workunit.client.0.smithi089.stdout:test ec_list on pid 80431 2024-03-21T13:14:12.583 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=ec_list 2024-03-21T13:14:12.583 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80431 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test ec_list on pid 80431' 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80431 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s misc 2024-03-21T13:14:12.584 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' misc' 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.585 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80200/exe 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80263/exe 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.586 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo misc 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.587 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.588 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.589 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_cls 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.590 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ cls: /' 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_cls.log 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.591 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.592 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.593 INFO:tasks.workunit.client.0.smithi089.stdout:test misc on pid 80481 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=misc 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80481 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test misc on pid 80481' 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80481 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80317/exe 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ handler_error: /' 2024-03-21T13:14:12.596 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.597 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s pool 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.598 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_handler_error 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_ec_io 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' pool' 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.599 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo pool 2024-03-21T13:14:12.600 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.600 INFO:tasks.workunit.client.0.smithi089.stdout:test pool on pid 80518 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=pool 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80518 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test pool on pid 80518' 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80518 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.603 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.604 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.605 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ ec_io: /' 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.606 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s read_operations 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.607 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' read_operations' 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo read_operations 2024-03-21T13:14:12.608 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.609 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.610 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.611 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80312/exe 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.612 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.613 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.614 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.615 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.616 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.617 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80481/exe 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.618 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.619 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_io 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.620 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.621 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.621 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.621 INFO:tasks.workunit.client.0.smithi089.stdout:test read_operations on pid 80593 2024-03-21T13:14:12.626 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.626 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.626 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.626 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.626 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=read_operations 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80593 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test read_operations on pid 80593' 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80593 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_misc.log 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.627 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ misc: /' 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.628 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_io.log 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ io: /' 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_misc 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s snapshots 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' snapshots' 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo snapshots 2024-03-21T13:14:12.629 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.630 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.631 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.632 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.633 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.634 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=snapshots 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80629 2024-03-21T13:14:12.635 INFO:tasks.workunit.client.0.smithi089.stdout:test snapshots on pid 80629 2024-03-21T13:14:12.639 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test snapshots on pid 80629' 2024-03-21T13:14:12.639 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80629 2024-03-21T13:14:12.639 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.639 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.639 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80361/exe 2024-03-21T13:14:12.639 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-21T13:14:12.639 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s watch_notify 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.640 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' watch_notify' 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.641 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.642 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo watch_notify 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.643 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_list 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.644 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.645 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=watch_notify 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80654 2024-03-21T13:14:12.646 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test watch_notify on pid 80654' 2024-03-21T13:14:12.647 INFO:tasks.workunit.client.0.smithi089.stdout:test watch_notify on pid 80654 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80654 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_list.log 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.651 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.652 INFO:tasks.workunit.client.0.smithi089.stderr:++ printf %25s write_operations 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=' write_operations' 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.653 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:++ awk '{print $1}' 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.654 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.655 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.656 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:++ echo write_operations 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.657 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.658 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.659 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:+ ff=write_operations 2024-03-21T13:14:12.660 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.661 INFO:tasks.workunit.client.0.smithi089.stdout:test write_operations on pid 80671 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80671 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 'test write_operations on pid 80671' 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ pids[$f]=80671 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ ret=0 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 1 -eq 1 ']' 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79523 2024-03-21T13:14:12.673 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79523 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80431/exe 2024-03-21T13:14:12.674 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.675 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -f /etc/bashrc ']' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+ . /etc/bashrc 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.676 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -z '' ']' 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ BASHRCSOURCED=Y 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.677 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:++ shopt -q login_shell 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80518/exe 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.678 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_ec_list 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ ec_list: /' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.679 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ list: /' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.680 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+++ umask 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.681 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' 0022 -eq 0 ']' 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ SHELL=/bin/bash 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.682 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_pool.log 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_pool 2024-03-21T13:14:12.683 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ pool: /' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80593/exe 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.684 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.685 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.686 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_read_operations 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.687 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80629/exe 2024-03-21T13:14:12.688 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.689 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.690 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ read_operations: /' 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ snapshots: /' 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.691 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.692 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_snapshots 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'grep=grep --color=auto' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorls.sh 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' '!' -t 0 ']' 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:+++ return 2024-03-21T13:14:12.693 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80654/exe 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export S_COLORS=auto 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:+++ S_COLORS=auto 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.694 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ watch_notify: /' 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-21T13:14:12.695 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_watch_notify 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-21T13:14:12.696 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/gawk.sh 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/lang.sh 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/locale 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-21T13:14:12.697 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:++++ export LANG=en_US.UTF-8 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:++++ LANG=en_US.UTF-8 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ LANG=en_US.UTF-8 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ unset LANG_backup config 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n '' ']' 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' dumb = linux ']' 2024-03-21T13:14:12.698 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/less.sh 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++ for i in /etc/profile.d/*.sh 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++ '[' '' ']' 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++ . /etc/profile.d/which2.sh 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:+++++ readlink /proc/80671/exe 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:++++ basename /usr/bin/bash 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_declare='declare -f' 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:+++ which_opt=-f 2024-03-21T13:14:12.699 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export which_declare 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+++ export -f which 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset i 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:++ unset -f pathmunge 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+ export PATH 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+ unset rc 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph_test_neorados_write_operations 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-21T13:14:12.700 INFO:tasks.workunit.client.0.smithi089.stderr:+ sed 's/^/ write_operations: /' 2024-03-21T13:14:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:12 smithi152 ceph-mon[34051]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1220612459' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-21T13:14:13.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:12 smithi089 ceph-mon[39209]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1220612459' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-21T13:14:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:12 smithi089 ceph-mon[30095]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1220612459' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-21T13:14:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/283280144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1160527341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:13 smithi152 ceph-mon[34051]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/283280144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1160527341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[30095]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/283280144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1160527341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:13 smithi089 ceph-mon[39209]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T13:14:17.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:14:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:14:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:18.057 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:14:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:14:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/963348730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2454213145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi089-80496-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2263924623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi089-79426-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/963348730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2454213145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T13:14:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi089-80496-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2263924623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi089-79426-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/963348730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2454213145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi089-80496-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2263924623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi089-79426-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2899270889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2605818796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/757143026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/217321201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2280427169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3042003945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3117250204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T13:14:20.404 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1901994884' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi089-79478-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.405 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3995710114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.405 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/994541235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi089-79952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.405 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2918010262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80167-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2899270889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2605818796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/757143026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/217321201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2280427169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3042003945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3117250204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1901994884' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi089-79478-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3995710114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/994541235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi089-79952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2918010262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80167-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi089-80681-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2899270889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2605818796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/757143026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/217321201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2280427169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3042003945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3117250204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1901994884' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi089-79478-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3995710114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/994541235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi089-79952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:20.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2918010262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80167-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: pgmap v75: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi089-80496-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2263924623' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi089-79426-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2654916417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2275704285' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi089-80496-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1901994884' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi089-79478-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3995710114' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/994541235' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi089-79952-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2918010262' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80167-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/947351343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:21 smithi152 ceph-mon[34051]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: pgmap v75: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi089-80496-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2263924623' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi089-79426-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2654916417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2275704285' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi089-80496-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1901994884' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi089-79478-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3995710114' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/994541235' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi089-79952-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2918010262' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80167-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: pgmap v75: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi089-80496-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi089-79447-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2263924623' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi089-79426-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2654916417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2275704285' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi089-80496-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1901994884' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi089-79478-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3995710114' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/994541235' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi089-79952-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2918010262' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80167-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi089-79462-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/947351343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[39209]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T13:14:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi089-79681-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/947351343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:21 smithi089 ceph-mon[30095]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] Global test environment set-up. 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosList.ListObjects (182 ms) 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (19 ms) 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset foo1,foo2,foo3 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo1 2024-03-21T13:14:21.759 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo2 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo3 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset foo1,foo4,foo5 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo4 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo5 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo1 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset foo6,foo7 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo7 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo6 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns1:foo4 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns1:foo5 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns2:foo7 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns2:foo6 2024-03-21T13:14:21.760 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns1:foo1 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: :foo1 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: :foo2 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: :foo3 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (60 ms) 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 1 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 10 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 13 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 7 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 14 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 0 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 15 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 11 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 5 0 2024-03-21T13:14:21.761 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 8 0 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 6 0 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 3 0 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 4 0 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 12 0 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 9 0 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 2 0 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (77 ms) 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: x cursor=MIN 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=1 cursor=10:02547ec2:::1:head 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=10 cursor=10:52ea6a34:::10:head 2024-03-21T13:14:21.762 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=13 cursor=10:566253c9:::13:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=7 cursor=10:5c6b0b28:::7:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=14 cursor=10:62a1935d:::14:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=0 cursor=10:6cac518f:::0:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=15 cursor=10:863748b0:::15:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=11 cursor=10:89d3ae78:::11:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=5 cursor=10:b29083e3:::5:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=8 cursor=10:bd63b0f1:::8:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=6 cursor=10:c4fdafeb:::6:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=3 cursor=10:cfc208b3:::3:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=4 cursor=10:d83876eb:::4:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=12 cursor=10:de5d7c5f:::12:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=9 cursor=10:e960b815:::9:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > oid=2 cursor=10:f905c69b:::2:head 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-21T13:14:21.763 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=1 cursor=10:02547ec2:::1:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=10 cursor=10:52ea6a34:::10:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=13 cursor=10:566253c9:::13:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:566253c9:::13:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=7 cursor=10:5c6b0b28:::7:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-03-21T13:14:21.764 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=14 cursor=10:62a1935d:::14:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=0 cursor=10:6cac518f:::0:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=15 cursor=10:863748b0:::15:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:863748b0:::15:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=11 cursor=10:89d3ae78:::11:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-03-21T13:14:21.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=5 cursor=10:b29083e3:::5:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=8 cursor=10:bd63b0f1:::8:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=6 cursor=10:c4fdafeb:::6:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=3 cursor=10:cfc208b3:::3:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-03-21T13:14:21.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=4 cursor=10:d83876eb:::4:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=12 cursor=10:de5d7c5f:::12:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=9 cursor=10:e960b815:::9:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:e960b815:::9:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : oid=2 cursor=10:f905c69b:::2:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-03-21T13:14:21.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:cfc208b3:::3:head expected=10:cfc208b3:::3:head 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=3 expected=3 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:bd63b0f1:::8:head expected=10:bd63b0f1:::8:head 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=8 expected=8 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:b29083e3:::5:head expected=10:b29083e3:::5:head 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=5 expected=5 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:89d3ae78:::11:head expected=10:89d3ae78:::11:head 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=11 expected=11 2024-03-21T13:14:21.816 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:52ea6a34:::10:head expected=10:52ea6a34:::10:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=10 expected=10 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:e960b815:::9:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:e960b815:::9:head expected=10:e960b815:::9:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=9 expected=9 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:de5d7c5f:::12:head expected=10:de5d7c5f:::12:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=12 expected=12 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:d83876eb:::4:head expected=10:d83876eb:::4:head 2024-03-21T13:14:21.817 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-03-21T13:14:22.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : e api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-21T13:14:22.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [----------] Global test environment set-up. 2024-03-21T13:14:22.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-21T13:14:22.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-21T13:14:22.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-03-21T13:14:22.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-21T13:14:22.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (4 ms) 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (5 ms) 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3 ms) 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4 ms) 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-21T13:14:22.755 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (4 ms) 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [----------] 12 tests from AsioRados (27 ms total) 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [----------] Global test environment tear-down 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (9953 ms total) 2024-03-21T13:14:22.756 INFO:tasks.workunit.client.0.smithi089.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-21T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: pgmap v76: 356 pgs: 189 creating+peering, 132 active+clean, 35 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: pgmap v78: 900 pgs: 189 creating+peering, 132 active+clean, 579 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi089-80496-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/340156203' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:22.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T13:14:22.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:22.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:22.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:22 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:22.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: pgmap v76: 356 pgs: 189 creating+peering, 132 active+clean, 35 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:22.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:22.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: pgmap v78: 900 pgs: 189 creating+peering, 132 active+clean, 579 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:22.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:22.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi089-80496-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:22.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/340156203' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: pgmap v76: 356 pgs: 189 creating+peering, 132 active+clean, 35 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: pgmap v78: 900 pgs: 189 creating+peering, 132 active+clean, 579 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi089-80496-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi089-79505-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi089-79539-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi089-80151-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi089-79897-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi089-79872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi089-79725-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:22.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/340156203' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:22.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T13:14:22.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:22.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:22.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:22 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:23.493 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [----------] Global test environment set-up. 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5063 ms) 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ OK ] LibRadosService.RegisterLate (27 ms) 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: cluster: 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: id: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: health: HEALTH_OK 2024-03-21T13:14:23.494 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: services: 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: data: 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: usage: 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: pgs: 132 active+clean 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.495 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: io: 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: cluster: 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: id: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: health: HEALTH_WARN 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 7 pool(s) do not have an application enabled 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: services: 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-21T13:14:23.496 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: data: 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: pools: 13 pools, 356 pgs 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: usage: 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: pgs: 9.831% pgs unknown 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 53.090% pgs not active 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 189 creating+peering 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 132 active+clean 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 35 unknown 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: io: 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.497 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: cluster: 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: id: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: health: HEALTH_WARN 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 24 pool(s) do not have an application enabled 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: services: 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-21T13:14:23.498 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-21T13:14:23.499 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-21T13:14:23.499 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:23.499 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: data: 2024-03-21T13:14:23.499 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: pools: 39 pools, 1188 pgs 2024-03-21T13:14:23.499 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: objects: 205 objects, 583 KiB 2024-03-21T13:14:23.994 INFO:tasks.workunit.client.0.smithi089.stdout: api_servic api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-21T13:14:23.994 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-21T13:14:23.994 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-21T13:14:23.995 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-21T13:14:23.995 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (11554 ms) 2024-03-21T13:14:23.995 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (11554 ms total) 2024-03-21T13:14:23.995 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: 2024-03-21T13:14:23.995 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-21T13:14:23.995 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (11554 ms total) 2024-03-21T13:14:23.995 INFO:tasks.workunit.client.0.smithi089.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-21T13:14:24.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: pgmap v81: 1188 pgs: 849 creating+peering, 281 active+clean, 58 unknown; 583 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2507222749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:24.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:24.324 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:24.324 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:24.324 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:24.324 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:23 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: pgmap v81: 1188 pgs: 849 creating+peering, 281 active+clean, 58 unknown; 583 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2507222749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: pgmap v81: 1188 pgs: 849 creating+peering, 281 active+clean, 58 unknown; 583 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi089-80681-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi089-79633-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi089-79876-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79959-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80022-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2507222749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:24.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:23 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:24.366 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_1_[80345]: starting. 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_1_[80345]: creating pool ceph_test_rados_list_parallel.smithi089-80290 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_1_[80345]: created object 0... 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_1_[80345]: created object 25... 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_1_[80345]: created object 49... 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_1_[80345]: finishing. 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_1_[80345]: shutting down. 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_2_[80346]: starting. 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_2_[80346]: listing objects. 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_2_[80346]: listed object 0... 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_2_[80346]: listed object 25... 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_2_[80346]: saw 50 objects 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_2_[80346]: shutting down. 2024-03-21T13:14:24.367 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_3_[81961]: starting. 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_3_[81961]: creating pool ceph_test_rados_list_parallel.smithi089-80290 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_3_[81961]: created object 0... 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_3_[81961]: created object 25... 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_3_[81961]: created object 49... 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_3_[81961]: finishing. 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_3_[81961]: shutting down. 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_4_[81962]: starting. 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_4_[81962]: listing objects. 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_4_[81962]: listed object 0... 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_4_[81962]: listed object 25... 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_4_[81962]: saw 50 objects 2024-03-21T13:14:24.368 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_4_[81962]: shutting down. 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_5_[81963]: starting. 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_5_[81963]: removed 25 objects... 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_5_[81963]: removed half of the objects 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_5_[81963]: removed 50 objects... 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_5_[81963]: removed 50 objects 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_5_[81963]: shutting down. 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_6_[82095]: starting. 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_6_[82095]: creating pool ceph_test_rados_list_parallel.smithi089-80290 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_6_[82095]: created object 0... 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_6_[82095]: created object 25... 2024-03-21T13:14:24.369 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_6_[82095]: created object 49... 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_6_[82095]: finishing. 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_6_[82095]: shutting down. 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_7_[82096]: starting. 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_7_[82096]: listing objects. 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_7_[82096]: listed object 0... 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_7_[82096]: listed object 25... 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_7_[82096]: listed object 50... 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_7_[82096]: saw 56 objects 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_7_[82096]: shutting down. 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.370 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_8_[82097]: starting. 2024-03-21T13:14:24.371 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_8_[82097]: added 25 objects... 2024-03-21T13:14:24.371 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_8_[82097]: added half of the objects 2024-03-21T13:14:24.371 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_8_[82097]: added 50 objects... 2024-03-21T13:14:24.371 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_8_[82097]: added 50 objects 2024-03-21T13:14:24.371 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_8_[82097]: shutting down. 2024-03-21T13:14:24.371 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.371 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_9_[82289]: starting. 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_9_[82289]: creating pool ceph_test_rados_list_parallel.smithi089-80290 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_9_[82289]: created object 0... 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_9_[82289]: created object 25... 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_9_[82289]: created object 49... 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_9_[82289]: finishing. 2024-03-21T13:14:24.554 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_9_[82289]: shutting down. 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: starting. 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: listing objects. 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: listed object 0... 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: listed object 25... 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: listed object 50... 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: listed object 75... 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: saw 96 objects 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_10_[82290]: shutting down. 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.555 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_13_[82295]: starting. 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_13_[82295]: removed 25 objects... 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_13_[82295]: removed half of the objects 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_13_[82295]: removed 50 objects... 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_13_[82295]: removed 50 objects 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_13_[82295]: shutting down. 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_11_[82292]: starting. 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_11_[82292]: added 25 objects... 2024-03-21T13:14:24.556 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_11_[82292]: added half of the objects 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_11_[82292]: added 50 objects... 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_11_[82292]: added 50 objects 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_11_[82292]: shutting down. 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_12_[82293]: starting. 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_12_[82293]: added 25 objects... 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_12_[82293]: added half of the objects 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_12_[82293]: added 50 objects... 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_12_[82293]: added 50 objects 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_12_[82293]: shutting down. 2024-03-21T13:14:24.557 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_14_[82404]: starting. 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_14_[82404]: creating pool ceph_test_rados_list_parallel.smithi089-80290 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_14_[82404]: created object 0... 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_14_[82404]: created object 25... 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_14_[82404]: created object 49... 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_14_[82404]: finishing. 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_14_[82404]: shutting down. 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.558 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.559 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.559 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:24.559 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: starting. 2024-03-21T13:14:24.559 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: listing objects. 2024-03-21T13:14:24.559 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: listed object 0... 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: listed object 2 handler_error: Running main() from gmock_main.cc 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [----------] Global test environment set-up. 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (12322 ms) 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [----------] 1 test from neocls_handler_error (12322 ms total) 2024-03-21T13:14:24.857 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: 2024-03-21T13:14:24.858 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [----------] Global test environment tear-down 2024-03-21T13:14:24.858 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [==========] 1 test from 1 test suite ran. (12323 ms total) 2024-03-21T13:14:24.858 INFO:tasks.workunit.client.0.smithi089.stdout: handler_error: [ PASSED ] 1 test. 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (83 ms) 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-21T13:14:25.440 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (366 ms) 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (12089 ms) 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:23.658779+0000 mon.a [INF] from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:23.658889+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:23.672488+0000 mon.b [INF] from='client.? 172.21.15.89:0/171741435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:23.969547+0000 mon.c [INF] from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:23.987894+0000 mon.a [INF] from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.026560+0000 mon.b [INF] from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.031409+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.032841+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.034171+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.036120+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.037392+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:25.441 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.038717+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:25.442 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.039461+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:25.442 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.040217+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:25.442 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.041889+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: pgmap v84: 1164 pgs: 714 creating+peering, 215 active+clean, 235 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1934063345' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/171741435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: 9.0 deep-scrub starts 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: 9.0 deep-scrub ok 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:25.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:25.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T13:14:25.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:25.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: pgmap v84: 1164 pgs: 714 creating+peering, 215 active+clean, 235 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:14:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1934063345' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/171741435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: pgmap v84: 1164 pgs: 714 creating+peering, 215 active+clean, 235 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1934063345' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi089-79426-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/171741435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]: dispatch 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: 9.0 deep-scrub starts 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: 9.0 deep-scrub ok 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: 9.0 deep-scrub starts 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: 9.0 deep-scrub ok 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: api_cmd_pp: Running main() from gmock_main.cc 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (55 ms) 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-21T13:14:25.934 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (68 ms) 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (13469 ms) 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (13592 ms total) 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (13592 ms total) 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-21T13:14:25.935 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (1392 ms) 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (8 ms) 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (14 ms) 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (22 ms) 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (7 ms) 2024-03-21T13:14:25.936 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (6 ms) 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (6 ms) 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (7 ms) 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (12 ms) 2024-03-21T13:14:25.937 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (14 ms) 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (9 ms) 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (6 ms) 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (1531 ms total) 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (13509 ms total) 2024-03-21T13:14:25.938 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout:5... 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: listed object 50... 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: listed object 75... 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: listed object 100... 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: listed object 125... 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: saw 150 objects 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_15_[82405]: shutting down. 2024-03-21T13:14:25.939 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_16_[82406]: starting. 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_16_[82406]: added 25 objects... 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_16_[82406]: added half of the objects 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_16_[82406]: added 50 objects... 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_16_[82406]: added 50 objects 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: process_16_[82406]: shutting down. 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.940 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.941 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.941 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******************************* 2024-03-21T13:14:25.941 INFO:tasks.workunit.client.0.smithi089.stdout: list_parallel: ******* SUCCESS ********** 2024-03-21T13:14:26.251 INFO:tasks.workunit.client.0.smithi089.stdout: 2024-03-21T13:14:24.042777+0000 mon.c [INF] from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.313051+0000 mon.a [INF] from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.324421+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.838482+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.840665+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.840925+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi089-79979-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.841002+0000 mon.a [INF] from='client.? 172.21.15.89:0/1083702561' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi089-80496-1"}]': finished 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.841065+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.847609+0000 mon.b [INF] from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:26.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.849848+0000 mon.b [INF] from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.921255+0000 mon.b [INF] from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:24.964487+0000 mon.b [INF] from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.086792+0000 mon.c [INF] from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.185712+0000 client.admin [INF] onexx 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.197036+0000 mon.a [INF] from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.676730+0000 mon.a [INF] from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.919760+0000 mon.a [INF] from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:26.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.919945+0000 mon.a [INF] from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.2 deep-scrub starts 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.2 deep-scrub ok 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: pgmap v86: 1068 pgs: 530 creating+peering, 207 active+clean, 331 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 229 B/s wr, 1 op/s 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.9 deep-scrub starts 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.9 deep-scrub ok 2024-03-21T13:14:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: pgmap v88: 756 pgs: 302 creating+peering, 173 active+clean, 281 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: onexx 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.4 deep-scrub starts 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.4 deep-scrub ok 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.8 deep-scrub starts 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: 9.8 deep-scrub ok 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2139236308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi089-79876-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:26 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.2 deep-scrub starts 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.2 deep-scrub ok 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: pgmap v86: 1068 pgs: 530 creating+peering, 207 active+clean, 331 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 229 B/s wr, 1 op/s 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.9 deep-scrub starts 2024-03-21T13:14:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.9 deep-scrub ok 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: pgmap v88: 756 pgs: 302 creating+peering, 173 active+clean, 281 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: onexx 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.4 deep-scrub starts 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.4 deep-scrub ok 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.8 deep-scrub starts 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: 9.8 deep-scrub ok 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2139236308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.2 deep-scrub starts 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.2 deep-scrub ok 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: pgmap v86: 1068 pgs: 530 creating+peering, 207 active+clean, 331 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 229 B/s wr, 1 op/s 2024-03-21T13:14:26.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.9 deep-scrub starts 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.9 deep-scrub ok 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: pgmap v88: 756 pgs: 302 creating+peering, 173 active+clean, 281 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: onexx 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.4 deep-scrub starts 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.4 deep-scrub ok 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.8 deep-scrub starts 2024-03-21T13:14:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: 9.8 deep-scrub ok 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2139236308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:26.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi089-79876-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi089-79876-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:26 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:27.771 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: 9.5 deep-scrub starts 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: 9.5 deep-scrub ok 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: 9.1 deep-scrub starts 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: 9.1 deep-scrub ok 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:27.772 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi089-79876-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi089-79681-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi089-79876-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:27.773 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:14:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: 9.5 deep-scrub starts 2024-03-21T13:14:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: 9.5 deep-scrub ok 2024-03-21T13:14:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-21T13:14:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: 9.1 deep-scrub starts 2024-03-21T13:14:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: 9.1 deep-scrub ok 2024-03-21T13:14:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi089-79876-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi089-79681-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi089-79876-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: 9.5 deep-scrub starts 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: 9.5 deep-scrub ok 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: 9.1 deep-scrub starts 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: 9.1 deep-scrub ok 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi089-79876-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi089-79681-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi089-79876-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:27.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:28.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:14:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:14:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: 9.6 deep-scrub starts 2024-03-21T13:14:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: 9.6 deep-scrub ok 2024-03-21T13:14:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: twoxx 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:28.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: 9.6 deep-scrub starts 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: 9.6 deep-scrub ok 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: twoxx 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: 9.6 deep-scrub starts 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: 9.6 deep-scrub ok 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: twoxx 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-21T13:14:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:28.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:28.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:28.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: 9.7 deep-scrub starts 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: 9.7 deep-scrub ok 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: pgmap v91: 812 pgs: 106 creating+peering, 450 active+clean, 256 unknown; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 44 MiB/s wr, 639 op/s 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T13:14:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi089-79681-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi089-79876-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:30.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T13:14:30.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi089-79539-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: 9.7 deep-scrub starts 2024-03-21T13:14:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: 9.7 deep-scrub ok 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: pgmap v91: 812 pgs: 106 creating+peering, 450 active+clean, 256 unknown; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 44 MiB/s wr, 639 op/s 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi089-79681-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi089-79876-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi089-79539-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: 9.7 deep-scrub starts 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: 9.7 deep-scrub ok 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: pgmap v91: 812 pgs: 106 creating+peering, 450 active+clean, 256 unknown; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 44 MiB/s wr, 639 op/s 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi089-79681-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:30.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi089-79876-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.080 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi089-79539-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:30.568 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.920670+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:30.568 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.934257+0000 mon.c [INF] from='client.? 172.21.15.89:0/3529501914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.568 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.934644+0000 mon.c [INF] from='client.? 172.21.15.89:0/2139236308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.568 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.938491+0000 mon.a [INF] from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:30.568 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.940954+0000 mon.a [INF] from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:30.568 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.943417+0000 mon.a [INF] from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi089-79876-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.961522+0000 mon.a [INF] from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.962868+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi089-79447-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.963170+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:25.966922+0000 mon.a [INF] from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:27.986495+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:27.986580+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi089-80496-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:27.986681+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi089-79447-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:27.991816+0000 mon.c [INF] from='client.? 172.21.15.89:0/2504639176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:27.992117+0000 mon.b [INF] from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.501179+0000 mon.a [INF] from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.501669+0000 mon.a [INF] from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-21T13:14:30.569 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.508061+0000 mon.a [INF] from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.755122+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.755773+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.756082+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.756736+0000 mon.b [INF] from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.759487+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.759724+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.760137+0000 mon.b [INF] from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:28.999970+0000 mon.a [INF] from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi089-79681-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.000063+0000 mon.a [INF] from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi089-79876-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.000149+0000 mon.a [INF] from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi089-79539-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.000224+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi089-80681-1"}]': finished 2024-03-21T13:14:30.570 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.000314+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi089-79462-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.012721+0000 mon.b [INF] from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.029151+0000 mon.b [INF] from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.280161+0000 mon.b [INF] from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.281220+0000 mon.a [INF] from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi089-79539-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.291974+0000 mon.b [INF] from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.787336+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.787751+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.068 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.788150+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:31.069 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:29.788795+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.069 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.037800+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:31.069 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.037938+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.069 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.038021+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.069 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.038082+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:31.069 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.043623+0000 mon.b [INF] from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:31.114 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.043838+0000 mon.b [INF] from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": " watch_notify: Running main() from gmock_main.cc 2024-03-21T13:14:31.114 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-21T13:14:31.114 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-21T13:14:31.114 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-21T13:14:31.114 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-21T13:14:31.114 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: handle_notify cookie 94562692653584 notify_id 343597383680 notifier_gid 14901 2024-03-21T13:14:31.114 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (11382 ms) 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: Trying... 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: handle_notify cookie 94562705349664 notify_id 356482285568 notifier_gid 15084 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: Waiting for 3.000000000s 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: Timed out. 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: Flushing... 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: Flushed... 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7120 ms) 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (18502 ms total) 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (18502 ms total) 2024-03-21T13:14:31.115 INFO:tasks.workunit.client.0.smithi089.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-21T13:14:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: pgmap v94: 892 pgs: 90 creating+peering, 370 active+clean, 432 unknown; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 546 op/s 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2552243849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: threexx 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-21T13:14:31.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: pgmap v94: 892 pgs: 90 creating+peering, 370 active+clean, 432 unknown; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 546 op/s 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2552243849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: threexx 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T13:14:31.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-21T13:14:31.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:31.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: pgmap v94: 892 pgs: 90 creating+peering, 370 active+clean, 432 unknown; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 546 op/s 2024-03-21T13:14:31.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi089-80496-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2552243849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:31.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:31.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: threexx 2024-03-21T13:14:31.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T13:14:31.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-21T13:14:31.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.098 INFO:tasks.workunit.client.0.smithi089.stdout: cls: Running main() from gmock_main.cc 2024-03-21T13:14:32.098 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-21T13:14:32.098 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [----------] Global test environment set-up. 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [ OK ] NeoRadosCls.DNE (11445 ms) 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (8124 ms) 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [----------] 2 tests from NeoRadosCls (19569 ms total) 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [----------] Global test environment tear-down 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [==========] 2 tests from 1 test suite ran. (19570 ms total) 2024-03-21T13:14:32.099 INFO:tasks.workunit.client.0.smithi089.stdout: cls: [ PASSED ] 2 tests. 2024-03-21T13:14:32.101 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_1_[80309]: starting. 2024-03-21T13:14:32.101 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_1_[80309]: creating pool ceph_test_rados_delete_pools_parallel.smithi089-80260 2024-03-21T13:14:32.101 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_1_[80309]: created object 0... 2024-03-21T13:14:32.101 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_1_[80309]: created object 25... 2024-03-21T13:14:32.101 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_1_[80309]: created object 49... 2024-03-21T13:14:32.101 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_1_[80309]: finishing. 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_1_[80309]: shutting down. 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_2_[80310]: starting. 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_2_[80310]: deleting pool ceph_test_rados_delete_pools_parallel.smithi089-80260 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_2_[80310]: shutting down. 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: ******************************* 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_3_[82187]: starting. 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_3_[82187]: creating pool ceph_test_rados_delete_pools_parallel.smithi089-80260 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_3_[82187]: created object 0... 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_3_[82187]: created object 25... 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_3_[82187]: created object 49... 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_3_[82187]: finishing. 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_3_[82187]: shutting down. 2024-03-21T13:14:32.102 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: ******************************* 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_5_[82189]: starting. 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_5_[82189]: listing objects. 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_5_[82189]: listed object 0... 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_5_[82189]: listed object 25... 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_5_[82189]: saw 50 objects 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_5_[82189]: shutting down. 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: ******************************* 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_4_[82188]: starting. 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_4_[82188]: deleting pool ceph_test_rados_delete_pools_parallel.smithi089-80260 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: process_4_[82188]: shutting down. 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: ******************************* 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: ******************************* 2024-03-21T13:14:32.103 INFO:tasks.workunit.client.0.smithi089.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-21T13:14:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-21T13:14:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi089-79539-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4087180489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi089-79539-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4087180489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi089-79539-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi089-79462-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T13:14:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4087180489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:32.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:32.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:32.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:32.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:32.607 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: Running main() from gmock_main.cc 2024-03-21T13:14:32.607 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-21T13:14:32.607 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [----------] Global test environment set-up. 2024-03-21T13:14:32.607 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-21T13:14:32.607 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-21T13:14:32.607 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9994 ms) 2024-03-21T13:14:32.607 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2365 ms) 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (7741 ms) 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [----------] 3 tests from NeoRadosCmd (20100 ms total) 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [----------] Global test environment tear-down 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [==========] 3 tests from 1 test suite ran. (20100 ms total) 2024-03-21T13:14:32.608 INFO:tasks.workunit.client.0.smithi089.stdout: cmd: [ PASSED ] 3 tests. 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_1_[80307]: starting. 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_1_[80307]: creating pool ceph_test_rados_open_pools_parallel.smithi089-80213 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_1_[80307]: created object 0... 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_1_[80307]: created object 25... 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_1_[80307]: created object 49... 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_1_[80307]: finishing. 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_1_[80307]: shutting down. 2024-03-21T13:14:32.609 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_2_[80308]: starting. 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_2_[80308]: rados_pool_create. 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_2_[80308]: rados_ioctx_create. 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_2_[80308]: shutting down. 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: ******************************* 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_3_[82477]: starting. 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_3_[82477]: creating pool ceph_test_rados_open_pools_parallel.smithi089-80213 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_3_[82477]: created object 0... 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_3_[82477]: created object 25... 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_3_[82477]: created object 49... 2024-03-21T13:14:32.610 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_3_[82477]: finishing. 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_3_[82477]: shutting down. 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: ******************************* 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_4_[82478]: starting. 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_4_[82478]: rados_pool_create. 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_4_[82478]: rados_ioctx_create. 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: process_4_[82478]: shutting down. 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: ******************************* 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: ******************************* 2024-03-21T13:14:32.611 INFO:tasks.workunit.client.0.smithi089.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-21T13:14:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: 9.3 deep-scrub starts 2024-03-21T13:14:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: 9.3 deep-scrub ok 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: pgmap v97: 876 pgs: 92 creating+peering, 704 active+clean, 80 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 35 KiB/s wr, 33 op/s 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: fourxx 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/983263116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2"}]: dispatch 2024-03-21T13:14:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T13:14:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: 9.3 deep-scrub starts 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: 9.3 deep-scrub ok 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: pgmap v97: 876 pgs: 92 creating+peering, 704 active+clean, 80 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 35 KiB/s wr, 33 op/s 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: fourxx 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/983263116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2"}]: dispatch 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: 9.3 deep-scrub starts 2024-03-21T13:14:33.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: 9.3 deep-scrub ok 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: pgmap v97: 876 pgs: 92 creating+peering, 704 active+clean, 80 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 35 KiB/s wr, 33 op/s 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: fourxx 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi089-80681-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/983263116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2"}]: dispatch 2024-03-21T13:14:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T13:14:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:33.810 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: seed 79478 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (1416 ms) 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (17 ms) 2024-03-21T13:14:33.811 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (9 ms) 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5 ms) 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (6 ms) 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (7 ms) 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (7 ms) 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-21T13:14:33.812 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (4 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (6 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (23 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (6 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-21T13:14:33.813 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (6 ms) 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (1542 ms total) 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (671 ms) 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (19 ms) 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-03-21T13:14:33.814 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-21T13:14:33.815 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-03-21T13:14:34.235 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN api_stat: Running main() from gmock_main.cc 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [----------] Global test environment set-up. 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStat.Stat (246 ms) 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6 ms) 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStat.StatNS (16 ms) 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-21T13:14:34.236 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [----------] 5 tests from LibRadosStat (273 ms total) 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1865 ms) 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (6 ms) 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (199 ms) 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3 ms) 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (2073 ms total) 2024-03-21T13:14:34.237 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: 2024-03-21T13:14:34.238 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [----------] Global test environment tear-down 2024-03-21T13:14:34.238 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (21920 ms total) 2024-03-21T13:14:34.238 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-21T13:14:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-21T13:14:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: pool 'PoolQuotaPP_smithi089-79447-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-21T13:14:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:34 smithi152 ceph-mon[34051]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: pool 'PoolQuotaPP_smithi089-79447-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[30095]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi089-79478-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]: dispatch 2024-03-21T13:14:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:34.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:34.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:34.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:34.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:34.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:34.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: pool 'PoolQuotaPP_smithi089-79447-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-21T13:14:34.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:34 smithi089 ceph-mon[39209]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:14:35.102 INFO:tasks.workunit.client.0.smithi089.stdout:LibRadosStatECPP_smithi089-79897-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:35.102 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.172737+0000 mon.b [INF] from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.102 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.324146+0000 mon.c [INF] from='client.? 172.21.15.89:0/2552243849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.102 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.324246+0000 mon.b [INF] from='client.? 172.21.15.89:0/2216103209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi089-79979-1"}]: dispatch 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.502418+0000 client.admin [INF] threexx 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.566964+0000 mon.a [INF] from='client.? 172.21.15.89:0/1967843460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: got: 2024-03-21T13:14:30.578062+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi089-79426-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (10196 ms) 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (22735 ms total) 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (22735 ms total) 2024-03-21T13:14:35.103 INFO:tasks.workunit.client.0.smithi089.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-21T13:14:35.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: pgmap v99: 804 pgs: 91 creating+peering, 505 active+clean, 208 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.9 KiB/s wr, 5 op/s 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2521485908' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T13:14:35.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3734997059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: pgmap v99: 804 pgs: 91 creating+peering, 505 active+clean, 208 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.9 KiB/s wr, 5 op/s 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:35.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2521485908' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: pgmap v99: 804 pgs: 91 creating+peering, 505 active+clean, 208 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.9 KiB/s wr, 5 op/s 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127392575' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi089-79876-7"}]': finished 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi089-79979-1"}]': finished 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi089-79580-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:35.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:35.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2521485908' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T13:14:35.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:35.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:35.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:14:35.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi089-80681-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3734997059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:35.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-4", "overlaypool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-21T13:14:35.582 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3734997059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:14:35.583 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.584 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:35.584 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:35.584 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:35.584 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:35.584 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:35.584 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:35.750 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: seed 79681 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (1120 ms) 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (49 ms) 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1210 ms) 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-21T13:14:35.751 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1068 ms) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (11 ms) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (6 ms) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (26 ms) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (7 ms) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (3497 ms total) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (878 ms) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (4 ms) 2024-03-21T13:14:35.752 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2032 ms) 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1100 ms) 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (4029 ms total) 2024-03-21T13:14:35.753 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (23492 ms total) 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: Running main() from gmock_main.cc 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [----------] Global test environment set-up. 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ OK ] NeoRadosPools.PoolList (8772 ms) 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2628 ms) 2024-03-21T13:14:35.754 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (1940 ms) 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5152 ms) 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2036 ms) 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2627 ms) 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [----------] 6 tests from NeoRadosPools (23155 ms total) 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [----------] Global test environment tear-down 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [==========] 6 tests from 1 test suite ran. (23155 ms total) 2024-03-21T13:14:35.755 INFO:tasks.workunit.client.0.smithi089.stdout: pool: [ PASSED ] 6 tests. 2024-03-21T13:14:35.758 INFO:tasks.workunit.client.0.smithi089.stdout: list: Running main() from gmock_main.cc 2024-03-21T13:14:35.758 INFO:tasks.workunit.client.0.smithi089.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-21T13:14:35.758 INFO:tasks.workunit.client.0.smithi089.stdout: list: [----------] Global test environment set-up. 2024-03-21T13:14:35.758 INFO:tasks.workunit.client.0.smithi089.stdout: list: [----------] 3 tests from NeoradosList 2024-03-21T13:14:35.758 INFO:tasks.workunit.client.0.smithi089.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-21T13:14:35.758 INFO:tasks.workunit.client.0.smithi089.stdout: list: [ OK ] NeoradosList.ListObjects (12303 ms) 2024-03-21T13:14:35.758 INFO:tasks.workunit.client.0.smithi089.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: [ OK ] NeoradosList.ListObjectsNS (5692 ms) 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: [ OK ] NeoradosList.ListObjectsMany (5184 ms) 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: [----------] 3 tests from NeoradosList (23179 ms total) 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: [----------] Global test environment tear-down 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: [==========] 3 tests from 1 test suite ran. (23179 ms total) 2024-03-21T13:14:35.759 INFO:tasks.workunit.client.0.smithi089.stdout: list: [ PASSED ] 3 tests. 2024-03-21T13:14:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: pgmap v103: 844 pgs: 18 creating+peering, 466 active+clean, 360 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T13:14:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3139877991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]: dispatch 2024-03-21T13:14:36.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]: dispatch 2024-03-21T13:14:36.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: pgmap v103: 844 pgs: 18 creating+peering, 466 active+clean, 360 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3139877991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: pgmap v103: 844 pgs: 18 creating+peering, 466 active+clean, 360 unknown; 145 MiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:36.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2096380567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi089-79681-10"}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi089-79426-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-4-cache", "mode": "writeback"}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3852509747' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:14:36.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3139877991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:36.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1664783583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:36.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:36.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:36.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:14:36.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]: dispatch 2024-03-21T13:14:36.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]: dispatch 2024-03-21T13:14:36.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]: dispatch 2024-03-21T13:14:36.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: Running main() from gmock_main.cc 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [----------] Global test environment set-up. 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (12310 ms) 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (11955 ms) 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (24265 ms total) 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [----------] Global test environment tear-down 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (24265 ms total) 2024-03-21T13:14:36.807 INFO:tasks.workunit.client.0.smithi089.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-21T13:14:37.796 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:14:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3139877991' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]': finished 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:37.797 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: pgmap v106: 780 pgs: 32 creating+peering, 85 creating+activating, 5 active+clean+snaptrim, 521 active+clean, 9 active+clean+snaptrim_wait, 128 unknown; 145 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-21T13:14:37.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T13:14:37.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:37.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]: dispatch 2024-03-21T13:14:37.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]: dispatch 2024-03-21T13:14:37.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:37.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:37.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout:] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5 ms) 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-03-21T13:14:37.833 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (5 ms) 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-21T13:14:37.834 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (763 ms total) 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (25655 ms total) 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: Running main() from gmock_main.cc 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [----------] Global test environment set-up. 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-21T13:14:37.835 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (44 ms) 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: no timeout :/ 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: no timeout :/ 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: no timeout :/ 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: no timeout :/ 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: no timeout :/ 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (35 ms) 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (6 ms) 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-03-21T13:14:37.836 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (1034 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (10 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (28 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (8 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.RmXattr (9 ms) 2024-03-21T13:14:37.837 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIo.XattrIter (4 ms) 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [----------] 14 tests from LibRadosIo (1191 ms total) 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1130 ms) 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (1558 ms) 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (10 ms) 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-21T13:14:37.838 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (5 ms) 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (12 ms) 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [----------] 10 tests from LibRadosIoEC (2738 ms total) 2024-03-21T13:14:37.839 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: 2024-03-21T13:14:37.840 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [----------] Global test environment tear-down 2024-03-21T13:14:37.840 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [==========] 24 tests from 2 test suites ran. (25663 ms total) 2024-03-21T13:14:37.840 INFO:tasks.workunit.client.0.smithi089.stdout: api_io: [ PASSED ] 24 tests. 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: seed 79897 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (1284 ms) 2024-03-21T13:14:37.855 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (101 ms) 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (2 ms) 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (17 ms) 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (1405 ms total) 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (664 ms) 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-21T13:14:37.856 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (3 ms) 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (12 ms) 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (679 ms total) 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (25530 ms total) 2024-03-21T13:14:37.857 INFO:tasks.workunit.client.0.smithi089.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-21T13:14:38.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:14:38.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:38.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3139877991' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]': finished 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: pgmap v106: 780 pgs: 32 creating+peering, 85 creating+activating, 5 active+clean+snaptrim, 521 active+clean, 9 active+clean+snaptrim_wait, 128 unknown; 145 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]: dispatch 2024-03-21T13:14:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]: dispatch 2024-03-21T13:14:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:38.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:38.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3139877991' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:38.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:38.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:38.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi089-79580-1","app":"app1","key":"key1"}]': finished 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi089-80496-2"}]': finished 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1416476209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/483560594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2411408809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: pgmap v106: 780 pgs: 32 creating+peering, 85 creating+activating, 5 active+clean+snaptrim, 521 active+clean, 9 active+clean+snaptrim_wait, 128 unknown; 145 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]: dispatch 2024-03-21T13:14:38.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]: dispatch 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: Running main() from gmock_main.cc 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [----------] Global test environment set-up. 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (167 ms) 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-21T13:14:38.981 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.LockShared (4 ms) 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1151 ms) 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1143 ms) 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (13 ms) 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.Unlock (8 ms) 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (11 ms) 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-21T13:14:38.982 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (40 ms) 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [----------] 8 tests from LibRadosLock (2537 ms total) 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (2680 ms) 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (7 ms) 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1017 ms) 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1053 ms) 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (42 ms) 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-21T13:14:38.983 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (6 ms) 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (4812 ms total) 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [----------] Global test environment tear-down 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (26756 ms total) 2024-03-21T13:14:38.984 INFO:tasks.workunit.client.0.smithi089.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-21T13:14:39.538 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:39.538 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:39.538 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]': finished 2024-03-21T13:14:39.538 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]': finished 2024-03-21T13:14:39.538 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:39.539 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]': finished 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]': finished 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:39.540 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:39.541 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:39.541 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi089-79952-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79979-6", "pg_num": 4}]': finished 2024-03-21T13:14:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-4"}]': finished 2024-03-21T13:14:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi089-79478-22"}]': finished 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi089-79897-7"}]': finished 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi089-79462-16"}]': finished 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]: dispatch 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2770867637' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]: dispatch 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:39.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1015981990' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:40.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3358308415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: pgmap v109: 720 pgs: 24 creating+activating, 5 active+clean+snaptrim, 478 active+clean, 9 active+clean+snaptrim_wait, 204 unknown; 145 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]': finished 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3662728414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]: dispatch 2024-03-21T13:14:40.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:40.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]: dispatch 2024-03-21T13:14:40.824 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T13:14:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1015981990' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3358308415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: pgmap v109: 720 pgs: 24 creating+activating, 5 active+clean+snaptrim, 478 active+clean, 9 active+clean+snaptrim_wait, 204 unknown; 145 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]': finished 2024-03-21T13:14:40.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3662728414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4275958846' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi089-79539-10"}]': finished 2024-03-21T13:14:40.833 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-4", "tierpool": "test-rados-api-smithi089-79633-4-cache"}]': finished 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3935965624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1015981990' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3358308415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: pgmap v109: 720 pgs: 24 creating+activating, 5 active+clean+snaptrim, 478 active+clean, 9 active+clean+snaptrim_wait, 204 unknown; 145 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]: dispatch 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.834 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi089-79426-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi089-80681-2"}]': finished 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "overlaypool": "test-rados-api-smithi089-79979-6"}]': finished 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3662728414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]: dispatch 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]: dispatch 2024-03-21T13:14:40.835 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi089-80681-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:41 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:14:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi089-80681-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi089-80681-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:14:42.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:42.147 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-21T13:14:42.147 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-21T13:14:42.147 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify_test_cb 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (1393 ms) 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94643227988944 err -107 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (38 ms) 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94643228008848 err -107 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1049 ms) 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-21T13:14:42.148 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 347892350978 cookie 94643228008848 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (31 ms) 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 347892350977 cookie 94643228040256 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (14 ms) 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 347892350976 cookie 94643228043728 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (48 ms) 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 347892350979 cookie 94643228043728 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 347892350979 cookie 94643228090816 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (13 ms) 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-21T13:14:42.149 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 347892350977 cookie 94643228043728 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 352187318274 cookie 94643228043728 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94643228043728 err -107 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_cb from 14781 notify_id 377957122049 cookie 94643228090816 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94643228090816 err -107 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11604 ms total) 2024-03-21T13:14:42.150 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: watch_notify_test_cb 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1228 ms) 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1228 ms total) 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-21T13:14:42.151 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (29804 ms total) 2024-03-21T13:14:42.160 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-21T13:14:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]': finished 2024-03-21T13:14:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi089-80681-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1943225755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: pgmap v112: 584 pgs: 64 unknown, 64 creating+peering, 456 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi089-80681-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T13:14:43.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3180861426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]': finished 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi089-80681-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1943225755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: pgmap v112: 584 pgs: 64 unknown, 64 creating+peering, 456 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T13:14:43.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi089-80681-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3180861426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79979-6", "mode": "writeback"}]': finished 2024-03-21T13:14:43.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi089-80681-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2182978248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1943225755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: pgmap v112: 584 pgs: 64 unknown, 64 creating+peering, 456 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi089-80681-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]: dispatch 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi089-79952-12"}]': finished 2024-03-21T13:14:43.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi089-79725-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:43.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T13:14:43.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3180861426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout:e: usage: 252 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: pgs: 4.882% pgs unknown 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 71.465% pgs not active 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 849 creating+peering 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 281 active+clean 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 58 unknown 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:43.614 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: io: 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: client: 3.0 KiB/s rd, 1.2 KiB/s wr, 2 op/s rd, 1 op/s wr 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6056 ms) 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ OK ] LibRadosService.Status (20076 ms) 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [----------] 4 tests from LibRadosService (31222 ms total) 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [----------] Global test environment tear-down 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31222 ms total) 2024-03-21T13:14:43.615 INFO:tasks.workunit.client.0.smithi089.stdout: api_service: [ PASSED ] 4 tests. 2024-03-21T13:14:43.727 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-21T13:14:43.727 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-21T13:14:43.727 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-21T13:14:43.727 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-21T13:14:43.727 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5060 ms) 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (1053 ms) 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20029 ms) 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: attempt 0 of 20 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: failed to deregister: 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: { 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "epoch": 19, 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "modified": "2024-03-21T13:14:39.097492+0000", 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "services": { 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "laundry": { 2024-03-21T13:14:43.728 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "daemons": { 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "summary": "", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "close-test-pid80056": { 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "start_epoch": 19, 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "start_stamp": "2024-03-21T13:14:38.600363+0000", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "gid": 26455, 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "addr": "172.21.15.89:0/2295219776", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "metadata": { 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "arch": "x86_64", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_release": "squid", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_version_short": "19.0.0-2267-g960fbbe4", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-21T13:14:43.729 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro": "centos", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro_version": "9", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "foo": "bar", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "hostname": "smithi089", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "kernel_version": "5.14.0-430.el9.x86_64", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "mem_total_kb": "32488116", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "os": "Linux", 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "this": "that" 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: }, 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "task_status": {} 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: }, 2024-03-21T13:14:43.730 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "pid80003": { 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "start_epoch": 18, 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "start_stamp": "2024-03-21T13:14:23.565476+0000", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "gid": 25052, 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "addr": "172.21.15.89:0/1298610940", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "metadata": { 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "arch": "x86_64", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_release": "squid", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_version_short": "19.0.0-2267-g960fbbe4", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro": "centos", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro_version": "9", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "foo": "bar", 2024-03-21T13:14:43.731 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "hostname": "smithi089", 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024", 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "kernel_version": "5.14.0-430.el9.x86_64", 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "mem_total_kb": "32488116", 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "os": "Linux", 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "this": "that" 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: }, 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "task_status": {} 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: } 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: } 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: }, 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "rgw": { 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "daemons": { 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "summary": "", 2024-03-21T13:14:43.732 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "14445": { 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "start_epoch": 12, 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "start_stamp": "2024-03-21T13:11:23.433076+0000", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "gid": 14445, 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "addr": "172.21.15.89:0/533721271", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "metadata": { 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "arch": "x86_64", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_release": "squid", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "ceph_version_short": "19.0.0-2267-g960fbbe4", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "container_hostname": "smithi089", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro": "centos", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "distro_version": "8", 2024-03-21T13:14:43.733 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "frontend_type#0": "beast", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "hostname": "smithi089", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "id": "foo.a", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "kernel_version": "5.14.0-430.el9.x86_64", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "mem_total_kb": "32488116", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "num_handles": "1", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "os": "Linux", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "pid": "7", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "realm_id": "", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "realm_name": "", 2024-03-21T13:14:43.734 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "zone_id": "b6653998-93ec-4f62-89de-720c082fb381", 2024-03-21T13:14:43.735 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "zone_name": "default", 2024-03-21T13:14:43.735 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "zonegroup_id": "86eda7d4-4367-4fe4-8439-2c7d0cc19266", 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "zonegroup_name": "defa api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: notify 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1197 ms) 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-21T13:14:44.309 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (2030 ms) 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (3227 ms total) 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: notify 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (148 ms) 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: notify 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5246 ms) 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (12 ms) 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-21T13:14:44.310 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (10 ms) 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924194010352 notify_id 420906795012 notifier_gid 25159 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (26 ms) 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924194010352 notify_id 420906795013 notifier_gid 25159 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (15 ms) 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195339520 notify_id 420906795009 notifier_gid 25159 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (16 ms) 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195357024 notify_id 420906795010 notifier_gid 25159 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (9 ms) 2024-03-21T13:14:44.311 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195357024 notify_id 420906795013 notifier_gid 25159 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (26 ms) 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195360304 notify_id 420906795010 notifier_gid 25159 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (35 ms) 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: trying... 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195372816 notify_id 420906795014 notifier_gid 25159 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: timed out 2024-03-21T13:14:44.312 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: flushing 2024-03-21T13:14:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: pgmap v114: 520 pgs: 160 unknown, 360 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-21T13:14:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi089-80681-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3778708134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/875120911' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:44 smithi152 ceph-mon[34051]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: pgmap v114: 520 pgs: 160 unknown, 360 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi089-80681-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3778708134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/875120911' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:45.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[30095]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: pgmap v114: 520 pgs: 160 unknown, 360 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi089-80681-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3778708134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/875120911' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:45.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:45.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:45.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:45.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:45.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:44 smithi089 ceph-mon[39209]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T13:14:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/772382293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2117051780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T13:14:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/772382293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2117051780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:46 smithi089 ceph-mon[30095]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/772382293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2117051780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:46.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:47.501 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: pgmap v117: 560 pgs: 232 unknown, 328 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:47.501 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:47.501 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T13:14:47.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2149884196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:47.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:47.502 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:14:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:47.502 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: pgmap v117: 560 pgs: 232 unknown, 328 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:47.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:47.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T13:14:47.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2149884196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:47.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: pgmap v117: 560 pgs: 232 unknown, 328 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi089-79426-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2149884196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:48.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:14:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:14:48.843 INFO:tasks.workunit.client.0.smithi089.stdout: ult" 2024-03-21T13:14:48.843 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: }, 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: "task_status": {} 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: } 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: } 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: } 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: } 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: } 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: attempt 1 of 20 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10298 ms) 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36440 ms total) 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-21T13:14:48.844 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36440 ms total) 2024-03-21T13:14:48.845 INFO:tasks.workunit.client.0.smithi089.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-21T13:14:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3726096812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T13:14:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3726096812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi089-79447-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3726096812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: pgmap v120: 592 pgs: 136 creating+peering, 96 unknown, 360 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:14:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2684506177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/341971771' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: pgmap v120: 592 pgs: 136 creating+peering, 96 unknown, 360 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2684506177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/341971771' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: pgmap v120: 592 pgs: 136 creating+peering, 96 unknown, 360 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-80086-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2684506177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/341971771' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: Running main() from gmock_main.cc 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [----------] Global test environment set-up. 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (16419 ms) 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (10951 ms) 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (11759 ms) 2024-03-21T13:14:51.729 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [----------] 3 tests from NeoradosECList (39129 ms total) 2024-03-21T13:14:51.730 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: 2024-03-21T13:14:51.730 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [----------] Global test environment tear-down 2024-03-21T13:14:51.730 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (39129 ms total) 2024-03-21T13:14:51.730 INFO:tasks.workunit.client.0.smithi089.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-21T13:14:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: pgmap v123: 680 pgs: 64 creating+peering, 288 unknown, 328 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:14:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2895239693' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: pgmap v123: 680 pgs: 64 creating+peering, 288 unknown, 328 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:14:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2895239693' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:52.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: pgmap v123: 680 pgs: 64 creating+peering, 288 unknown, 328 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi089-79872-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi089-80681-3"}]: dispatch 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:52.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2895239693' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:52.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:52.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:52.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: pgmap v125: 552 pgs: 32 creating+peering, 520 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-21T13:14:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]': finished 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/181441880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1552773290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi089-79447-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1552773290' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi089-79447-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:52 smithi152 ceph-mon[34051]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: pgmap v125: 552 pgs: 32 creating+peering, 520 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]': finished 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/181441880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1552773290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi089-79447-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1552773290' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi089-79447-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:53.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[30095]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: pgmap v125: 552 pgs: 32 creating+peering, 520 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4172619617' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi089-80681-3"}]': finished 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-7"}]': finished 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi089-79426-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/181441880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1552773290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi089-79447-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:14:53.093 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1552773290' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi089-79447-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.094 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:53.094 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:14:53.094 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:53.094 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:52 smithi089 ceph-mon[39209]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T13:14:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: pgmap v128: 680 pgs: 192 unknown, 488 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-21T13:14:54.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]: dispatch 2024-03-21T13:14:54.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]: dispatch 2024-03-21T13:14:54.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]': finished 2024-03-21T13:14:54.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]': finished 2024-03-21T13:14:54.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T13:14:54.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:54.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: pgmap v128: 680 pgs: 192 unknown, 488 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]: dispatch 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]: dispatch 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]': finished 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]': finished 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:54.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]: dispatch 2024-03-21T13:14:54.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: pgmap v128: 680 pgs: 192 unknown, 488 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]: dispatch 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]: dispatch 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-7"}]': finished 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4"}]': finished 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:54.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]: dispatch 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: 2024-03-21T13:14:55.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (12471 ms) 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (5707 ms) 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2552 ms) 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (4723 ms) 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4656 ms) 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3950 ms) 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-21T13:14:55.645 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (9195 ms) 2024-03-21T13:14:55.646 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (43254 ms total) 2024-03-21T13:14:55.646 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: 2024-03-21T13:14:55.646 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-21T13:14:55.646 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (43254 ms total) 2024-03-21T13:14:55.646 INFO:tasks.workunit.client.0.smithi089.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-21T13:14:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3655684314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:55.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:55.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:55.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:14:55.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]': finished 2024-03-21T13:14:55.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:55.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T13:14:55.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3655684314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:55.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:14:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]': finished 2024-03-21T13:14:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T13:14:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.829 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3655684314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:55.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:14:55.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:14:55.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi089-79979-4", "tierpool": "test-rados-api-smithi089-79979-6"}]': finished 2024-03-21T13:14:55.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi089-79426-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:55.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T13:14:55.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/584369144' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: flushed 2024-03-21T13:14:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3014 ms) 2024-03-21T13:14:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-21T13:14:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: trying... 2024-03-21T13:14:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195372816 notify_id 433791696899 notifier_gid 25159 2024-03-21T13:14:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: timed out 2024-03-21T13:14:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: flushing 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: flushed 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3015 ms) 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: List watches 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: notify2 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195372816 notify_id 442381631492 notifier_gid 25159 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: notify2 done 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: watch_check 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: unwatch2 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: flushing 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: done 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3040 ms) 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: List watches 2024-03-21T13:14:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: notify2 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: handle_notify cookie 93924195360304 notify_id 455266533379 notifier_gid 25159 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: notify2 done 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: watch_check 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: unwatch2 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: flushing 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: done 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3026 ms) 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (17639 ms total) 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-21T13:14:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (44430 ms total) 2024-03-21T13:14:56.787 INFO:tasks.workunit.client.0.smithi089.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-21T13:14:56.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:56.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: 9.6 deep-scrub starts 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: 9.6 deep-scrub ok 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: 9.3 deep-scrub starts 2024-03-21T13:14:56.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: 9.3 deep-scrub ok 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: 9.0 deep-scrub starts 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: 9.0 deep-scrub ok 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3901197724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.823 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:56.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: 9.6 deep-scrub starts 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: 9.6 deep-scrub ok 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: 9.3 deep-scrub starts 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: 9.3 deep-scrub ok 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: 9.0 deep-scrub starts 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: 9.0 deep-scrub ok 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3901197724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-03-21T13:14:56.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: 9.6 deep-scrub starts 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: 9.6 deep-scrub ok 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: 9.3 deep-scrub starts 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: 9.3 deep-scrub ok 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: 9.0 deep-scrub starts 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: 9.0 deep-scrub ok 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi089-79979-6", "pool2": "test-rados-api-smithi089-79979-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3901197724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:56.832 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: pgmap v131: 648 pgs: 192 unknown, 456 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.9 deep-scrub starts 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.9 deep-scrub ok 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.2 deep-scrub starts 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.2 deep-scrub ok 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.5 deep-scrub starts 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.5 deep-scrub ok 2024-03-21T13:14:57.801 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.8 deep-scrub starts 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.4 deep-scrub starts 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.8 deep-scrub ok 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: 9.4 deep-scrub ok 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3662766009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.802 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: Running main() from gmock_main.cc 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [----------] Global test environment set-up. 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (11371 ms) 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-21T13:14:57.821 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (6334 ms) 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (5417 ms) 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (4221 ms) 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (5209 ms) 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (4606 ms) 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (8030 ms) 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (45188 ms total) 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [----------] Global test environment tear-down 2024-03-21T13:14:57.822 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (45188 ms total) 2024-03-21T13:14:57.823 INFO:tasks.workunit.client.0.smithi089.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-21T13:14:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: pgmap v131: 648 pgs: 192 unknown, 456 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.9 deep-scrub starts 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.9 deep-scrub ok 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.2 deep-scrub starts 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.2 deep-scrub ok 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.5 deep-scrub starts 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.5 deep-scrub ok 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.8 deep-scrub starts 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.4 deep-scrub starts 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.8 deep-scrub ok 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: 9.4 deep-scrub ok 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:57.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3662766009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:57.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T13:14:57.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.827 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:14:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: pgmap v131: 648 pgs: 192 unknown, 456 active+clean; 145 MiB data, 735 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.9 deep-scrub starts 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.9 deep-scrub ok 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.2 deep-scrub starts 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.2 deep-scrub ok 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.5 deep-scrub starts 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.5 deep-scrub ok 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.8 deep-scrub starts 2024-03-21T13:14:57.830 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.4 deep-scrub starts 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.8 deep-scrub ok 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: 9.4 deep-scrub ok 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi089-79447-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3662766009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:57.831 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:58.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:14:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:14:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: pgmap v134: 612 pgs: 96 unknown, 64 creating+peering, 452 active+clean; 217 MiB data, 999 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T13:14:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3066359209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: pgmap v134: 612 pgs: 96 unknown, 64 creating+peering, 452 active+clean; 217 MiB data, 999 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3066359209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: pgmap v134: 612 pgs: 96 unknown, 64 creating+peering, 452 active+clean; 217 MiB data, 999 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3066359209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:58.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:58.942 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-21T13:14:58.942 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] Global test environment set-up. 2024-03-21T13:14:58.942 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: 2024-03-21T13:14:12.298+0000 7f6f5f039780 -1 failed for service _ceph-mon._tcp 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: 2024-03-21T13:14:12.298+0000 7f6f5f039780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (63 ms) 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2011 ms) 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2074 ms total) 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-21T13:14:58.943 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: started 0x563b94886600 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: started 0x563b94bb8000 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: started 2 aios 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: waiting 0x563b94886600 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: waiting 0x563b94bb8000 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: done. 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (11625 ms) 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (11625 ms total) 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.Exec (100 ms) 2024-03-21T13:14:58.944 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (15 ms) 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6 ms) 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6630 ms) 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21689 ms) 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28440 ms total) 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [----------] Global test environment tear-down 2024-03-21T13:14:58.945 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (46685 ms total) 2024-03-21T13:14:58.946 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-21T13:14:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: 9.1 deep-scrub starts 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: 9.1 deep-scrub ok 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3906967863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:14:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: 9.1 deep-scrub starts 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: 9.1 deep-scrub ok 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3906967863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:14:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: 9.1 deep-scrub starts 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: 9.1 deep-scrub ok 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi089-79426-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3906967863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:14:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:14:59.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:14:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: pgmap v137: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 999 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]': finished 2024-03-21T13:15:00.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:00.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:15:00.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]: dispatch 2024-03-21T13:15:00.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]: dispatch 2024-03-21T13:15:00.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: pgmap v137: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 999 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]': finished 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]: dispatch 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]: dispatch 2024-03-21T13:15:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: pgmap v137: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 999 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-9"}]': finished 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi089-79447-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]: dispatch 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]: dispatch 2024-03-21T13:15:00.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:01 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]': finished 2024-03-21T13:15:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:01 smithi152 ceph-mon[34051]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:15:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3789828684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi089-79426-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]': finished 2024-03-21T13:15:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[30095]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:15:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3789828684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi089-79426-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:02.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-9", "mode": "writeback"}]': finished 2024-03-21T13:15:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[39209]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:15:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3789828684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi089-79426-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: pgmap v140: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3789828684' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi089-79426-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3568267985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: pgmap v140: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3789828684' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi089-79426-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3568267985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: pgmap v140: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3789828684' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi089-79426-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3568267985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3568267985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:15:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T13:15:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi089-79725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:15:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3568267985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi089-79725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3568267985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79688-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi089-79725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]: dispatch 2024-03-21T13:15:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: pgmap v143: 452 pgs: 96 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]': finished 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: osdmap e119: 8 total, 8 up, 8 in 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]: dispatch 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2309794604' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]: dispatch 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:05.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi089-79725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:05.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:05.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]': finished 2024-03-21T13:15:05.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T13:15:05.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2245543569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: pgmap v143: 452 pgs: 96 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-21T13:15:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:15:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]': finished 2024-03-21T13:15:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: osdmap e119: 8 total, 8 up, 8 in 2024-03-21T13:15:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]: dispatch 2024-03-21T13:15:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2309794604' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]: dispatch 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi089-79725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]': finished 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2245543569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: pgmap v143: 452 pgs: 96 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-9"}]': finished 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: osdmap e119: 8 total, 8 up, 8 in 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]: dispatch 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2309794604' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]: dispatch 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi089-79725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi089-79426-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache", "force_nonempty":""}]': finished 2024-03-21T13:15:05.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T13:15:05.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2245543569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:05.591 INFO:tasks.workunit.client.0.smithi089.stdout: misc: Running main() from gmock_main.cc 2024-03-21T13:15:05.591 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-21T13:15:05.591 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [----------] Global test environment set-up. 2024-03-21T13:15:05.591 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-21T13:15:05.591 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-21T13:15:05.591 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.Version (9933 ms) 2024-03-21T13:15:05.591 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2369 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.LongName (7744 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3143 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (4209 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4836 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.Exec (4978 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3979 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (4048 ms) 2024-03-21T13:15:05.592 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2779 ms) 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2032 ms) 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2964 ms) 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [----------] 12 tests from NeoRadosMisc (53015 ms total) 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [----------] Global test environment tear-down 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [==========] 12 tests from 1 test suite ran. (53015 ms total) 2024-03-21T13:15:05.593 INFO:tasks.workunit.client.0.smithi089.stdout: misc: [ PASSED ] 12 tests. 2024-03-21T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T13:15:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]: dispatch 2024-03-21T13:15:06.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]: dispatch 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]: dispatch 2024-03-21T13:15:06.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]: dispatch 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi089-79447-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79688-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/824524826' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]: dispatch 2024-03-21T13:15:06.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]: dispatch 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: pgmap v146: 556 pgs: 232 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]': finished 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1707770474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:07.812 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:15:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: pgmap v146: 556 pgs: 232 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]': finished 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1707770474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: pgmap v146: 556 pgs: 232 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79688-10", "tierpool":"test-rados-api-smithi089-79688-10-cache"}]': finished 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1707770474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:08.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:15:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: Running main() from gmock_main.cc 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [----------] Global test environment set-up. 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolList (9408 ms) 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (7389 ms) 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5692 ms) 2024-03-21T13:15:08.596 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3111 ms) 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4666 ms) 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (7292 ms) 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5807 ms) 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4956 ms) 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8025 ms) 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [----------] 9 tests from LibRadosPools (56346 ms total) 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: 2024-03-21T13:15:08.597 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [----------] Global test environment tear-down 2024-03-21T13:15:08.598 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (56346 ms total) 2024-03-21T13:15:08.598 INFO:tasks.workunit.client.0.smithi089.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-21T13:15:08.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: 9.7 deep-scrub starts 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: 9.7 deep-scrub ok 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: pgmap v149: 460 pgs: 64 unknown, 32 creating+peering, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2639620387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi089-79447-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: 9.7 deep-scrub starts 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: 9.7 deep-scrub ok 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: pgmap v149: 460 pgs: 64 unknown, 32 creating+peering, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2639620387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi089-79447-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:08.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: 9.7 deep-scrub starts 2024-03-21T13:15:08.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: 9.7 deep-scrub ok 2024-03-21T13:15:08.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: pgmap v149: 460 pgs: 64 unknown, 32 creating+peering, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-21T13:15:08.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi089-79426-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:08.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T13:15:08.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2639620387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi089-79447-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:08.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:08.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:08.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:09.990 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2639620387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi089-79447-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: osdmap e124: 8 total, 8 up, 8 in 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]': finished 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]: dispatch 2024-03-21T13:15:09.991 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]: dispatch 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2443384101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi089-79426-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2639620387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi089-79447-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: osdmap e124: 8 total, 8 up, 8 in 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]': finished 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]: dispatch 2024-03-21T13:15:09.992 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]: dispatch 2024-03-21T13:15:09.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2443384101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi089-79426-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2639620387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi089-79447-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: osdmap e124: 8 total, 8 up, 8 in 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-11"}]': finished 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]: dispatch 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]: dispatch 2024-03-21T13:15:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2443384101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi089-79426-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:10.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: pgmap v152: 428 pgs: 96 unknown, 32 creating+peering, 300 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-21T13:15:10.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:10.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:10.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]': finished 2024-03-21T13:15:10.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2443384101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi089-79426-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:10.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T13:15:10.974 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1951084419' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: pgmap v152: 428 pgs: 96 unknown, 32 creating+peering, 300 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]': finished 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2443384101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi089-79426-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1951084419' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:10.975 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: pgmap v152: 428 pgs: 96 unknown, 32 creating+peering, 300 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-21T13:15:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:11.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-11", "mode": "writeback"}]': finished 2024-03-21T13:15:11.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2443384101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi089-79426-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:11.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T13:15:11.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1951084419' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:11.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:11 smithi152 ceph-mon[34051]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T13:15:12.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:12.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:12.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:11 smithi089 ceph-mon[39209]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T13:15:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi089-79447-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:11 smithi089 ceph-mon[30095]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T13:15:12.582 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: Running main() from gmock_main.cc 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [----------] Global test environment set-up. 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (12284 ms) 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (7218 ms) 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3649 ms) 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-21T13:15:12.583 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3215 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3521 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3968 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3339 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3980 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (4043 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2763 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3290 ms) 2024-03-21T13:15:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2731 ms) 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2980 ms) 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3005 ms) 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (59986 ms total) 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [----------] Global test environment tear-down 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (59987 ms total) 2024-03-21T13:15:12.585 INFO:tasks.workunit.client.0.smithi089.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-21T13:15:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: pgmap v155: 492 pgs: 64 creating+peering, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 262 KiB/s wr, 13 op/s 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/653649612' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: pgmap v155: 492 pgs: 64 creating+peering, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 262 KiB/s wr, 13 op/s 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T13:15:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/653649612' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: pgmap v155: 492 pgs: 64 creating+peering, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 262 KiB/s wr, 13 op/s 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T13:15:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:13.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]: dispatch 2024-03-21T13:15:13.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/653649612' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:13.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:13 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:15:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]': finished 2024-03-21T13:15:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:13 smithi152 ceph-mon[34051]: osdmap e129: 8 total, 8 up, 8 in 2024-03-21T13:15:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1133656162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]': finished 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[30095]: osdmap e129: 8 total, 8 up, 8 in 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1133656162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:14.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:15:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-11"}]': finished 2024-03-21T13:15:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi089-79426-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[39209]: osdmap e129: 8 total, 8 up, 8 in 2024-03-21T13:15:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1133656162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: Running main() from gmock_main.cc 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [----------] Global test environment set-up. 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.Limits (12303 ms) 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (5706 ms) 2024-03-21T13:15:14.584 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.ReadOp (5184 ms) 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.SparseRead (4223 ms) 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3266 ms) 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (4385 ms) 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (4144 ms) 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3829 ms) 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-21T13:15:14.585 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3319 ms) 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.Trunc (3692 ms) 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.Remove (2969 ms) 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3014 ms) 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.RmXattr (2985 ms) 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3003 ms) 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: [----------] 14 tests from NeoRadosIo (62023 ms total) 2024-03-21T13:15:14.586 INFO:tasks.workunit.client.0.smithi089.stdout: io: 2024-03-21T13:15:14.587 INFO:tasks.workunit.client.0.smithi089.stdout: io: [----------] Global test environment tear-down 2024-03-21T13:15:14.587 INFO:tasks.workunit.client.0.smithi089.stdout: io: [==========] 14 tests from 1 test suite ran. (62023 ms total) 2024-03-21T13:15:14.587 INFO:tasks.workunit.client.0.smithi089.stdout: io: [ PASSED ] 14 tests. 2024-03-21T13:15:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:15 smithi152 ceph-mon[34051]: pgmap v158: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 262 KiB/s wr, 11 op/s 2024-03-21T13:15:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:15 smithi152 ceph-mon[34051]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:15.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:15 smithi152 ceph-mon[34051]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T13:15:15.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[39209]: pgmap v158: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 262 KiB/s wr, 11 op/s 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[39209]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[39209]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[30095]: pgmap v158: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 262 KiB/s wr, 11 op/s 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[30095]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:15.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi089-79447-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:15.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[30095]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T13:15:15.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: pgmap v161: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T13:15:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/465057535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: pgmap v161: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/465057535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: pgmap v161: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T13:15:16.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/465057535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]: dispatch 2024-03-21T13:15:16.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:16.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:17.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:15:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3003888815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi089-79447-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3003888815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi089-79447-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:17.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:18.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2418344340' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi089-79725-10"}]': finished 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi089-79426-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3003888815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi089-79447-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:18.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:18 smithi152 ceph-mon[34051]: pgmap v164: 420 pgs: 32 unknown, 96 creating+peering, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3003888815' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi089-79447-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:18 smithi152 ceph-mon[34051]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T13:15:19.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:19.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:19.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[39209]: pgmap v164: 420 pgs: 32 unknown, 96 creating+peering, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3003888815' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi089-79447-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[39209]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T13:15:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:19.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[30095]: pgmap v164: 420 pgs: 32 unknown, 96 creating+peering, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:19.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3003888815' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi089-79447-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:19.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:19.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[30095]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T13:15:19.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:19.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:19.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: osdmap e134: 8 total, 8 up, 8 in 2024-03-21T13:15:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/845934165' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi089-79426-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: pgmap v167: 388 pgs: 32 unknown, 64 creating+peering, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:15:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/845934165' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi089-79426-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T13:15:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1615999749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: osdmap e134: 8 total, 8 up, 8 in 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/845934165' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi089-79426-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: pgmap v167: 388 pgs: 32 unknown, 64 creating+peering, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/845934165' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi089-79426-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1615999749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: osdmap e134: 8 total, 8 up, 8 in 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/845934165' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi089-79426-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: pgmap v167: 388 pgs: 32 unknown, 64 creating+peering, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi089-79725-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/845934165' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi089-79426-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1615999749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:20.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:20 smithi152 ceph-mon[34051]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T13:15:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:20 smithi089 ceph-mon[39209]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T13:15:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi089-79447-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:21.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:20 smithi089 ceph-mon[30095]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T13:15:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: pgmap v170: 364 pgs: 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 127 KiB/s wr, 126 op/s 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3834700195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: pgmap v170: 364 pgs: 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 127 KiB/s wr, 126 op/s 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3834700195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:22.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:22.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:22.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:15:22.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: pgmap v170: 364 pgs: 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 127 KiB/s wr, 126 op/s 2024-03-21T13:15:22.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:15:22.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T13:15:22.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:22.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:22.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3834700195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:22.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]': finished 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]': finished 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]: dispatch 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]: dispatch 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:15:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3664530658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi089-79447-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]': finished 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]': finished 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]: dispatch 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]: dispatch 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:15:23.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3664530658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi089-79447-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-13"}]': finished 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi089-79426-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]': finished 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:15:23.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]: dispatch 2024-03-21T13:15:23.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:15:23.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3664530658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi089-79447-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:23 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:24.351 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]': finished 2024-03-21T13:15:24.351 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]': finished 2024-03-21T13:15:24.351 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3664530658' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi089-79447-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:24.351 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:24.351 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:23 smithi152 ceph-mon[34051]: osdmap e139: 8 total, 8 up, 8 in 2024-03-21T13:15:24.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:24.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]': finished 2024-03-21T13:15:24.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]': finished 2024-03-21T13:15:24.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3664530658' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi089-79447-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:24.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:24.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[30095]: osdmap e139: 8 total, 8 up, 8 in 2024-03-21T13:15:24.352 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:15:24.353 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-13", "mode": "writeback"}]': finished 2024-03-21T13:15:24.353 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]': finished 2024-03-21T13:15:24.353 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3664530658' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi089-79447-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:24.353 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:24.353 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:23 smithi089 ceph-mon[39209]: osdmap e139: 8 total, 8 up, 8 in 2024-03-21T13:15:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: pgmap v173: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 KiB/s wr, 126 op/s 2024-03-21T13:15:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.0"}]: dispatch 2024-03-21T13:15:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.1"}]: dispatch 2024-03-21T13:15:25.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.2"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.3"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.4"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.5"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.6"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.7"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.8"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.9"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: 176.5 scrub starts 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: 176.5 scrub ok 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]': finished 2024-03-21T13:15:25.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T13:15:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2363989398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2"}]: dispatch 2024-03-21T13:15:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: pgmap v173: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 KiB/s wr, 126 op/s 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.0"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.1"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.2"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.3"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.4"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.5"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.6"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.7"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.8"}]: dispatch 2024-03-21T13:15:25.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.9"}]: dispatch 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: 176.5 scrub starts 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: 176.5 scrub ok 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]': finished 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:25.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: pgmap v173: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 KiB/s wr, 126 op/s 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.0"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.1"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.2"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.3"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.4"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.5"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.6"}]: dispatch 2024-03-21T13:15:25.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.7"}]: dispatch 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.8"}]: dispatch 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "176.9"}]: dispatch 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: 176.5 scrub starts 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: 176.5 scrub ok 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pg_num","val":"11"}]': finished 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T13:15:25.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2363989398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2"}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2363989398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2"}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:25.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:26.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.0"}]: dispatch 2024-03-21T13:15:26.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.1"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.2"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.3"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.4"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.5"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.6"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.7"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.8"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.9"}]: dispatch 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.0 scrub starts 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.0 scrub ok 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.7 scrub starts 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.7 scrub ok 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.3 scrub starts 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.3 scrub ok 2024-03-21T13:15:26.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.8 scrub starts 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.8 scrub ok 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.2 deep-scrub starts 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: 176.2 deep-scrub ok 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: pgmap v176: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4006417937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.0"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.1"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.2"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.3"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.4"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.5"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.6"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.7"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.8"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.9"}]: dispatch 2024-03-21T13:15:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.0 scrub starts 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.0 scrub ok 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.7 scrub starts 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.7 scrub ok 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.3 scrub starts 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.3 scrub ok 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.8 scrub starts 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.8 scrub ok 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.2 deep-scrub starts 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: 176.2 deep-scrub ok 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: pgmap v176: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:26.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4006417937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.0"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.1"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.2"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.3"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.4"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.5"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.6"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.7"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.8"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "176.9"}]: dispatch 2024-03-21T13:15:26.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.0 scrub starts 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.0 scrub ok 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.7 scrub starts 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.7 scrub ok 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.3 scrub starts 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.3 scrub ok 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.8 scrub starts 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.8 scrub ok 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.2 deep-scrub starts 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: 176.2 deep-scrub ok 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: pgmap v176: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi089-79426-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:26.580 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi089-79628-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T13:15:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4006417937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:26.581 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: 176.6 scrub starts 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: 176.6 scrub ok 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: 176.4 deep-scrub starts 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: 176.4 deep-scrub ok 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-21T13:15:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: 176.6 scrub starts 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: 176.6 scrub ok 2024-03-21T13:15:27.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: 176.4 deep-scrub starts 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: 176.4 deep-scrub ok 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:15:27.499 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: 176.6 scrub starts 2024-03-21T13:15:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: 176.6 scrub ok 2024-03-21T13:15:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: 176.4 deep-scrub starts 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: 176.4 deep-scrub ok 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:15:27.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:15:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:15:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:28.095 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:15:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:15:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: 176.1 scrub starts 2024-03-21T13:15:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: 176.1 scrub ok 2024-03-21T13:15:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: pgmap v179: 396 pgs: 64 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 66 KiB/s rd, 12 KiB/s wr, 95 op/s 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: 176.9 scrub starts 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: 176.9 scrub ok 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 2 stray daemon(s) not managed by cephadm) 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1163072488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: 176.1 scrub starts 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: 176.1 scrub ok 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: pgmap v179: 396 pgs: 64 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 66 KiB/s rd, 12 KiB/s wr, 95 op/s 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: 176.9 scrub starts 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: 176.9 scrub ok 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 2 stray daemon(s) not managed by cephadm) 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1163072488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: 176.1 scrub starts 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: 176.1 scrub ok 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: pgmap v179: 396 pgs: 64 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 66 KiB/s rd, 12 KiB/s wr, 95 op/s 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: 176.9 scrub starts 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: 176.9 scrub ok 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 2 stray daemon(s) not managed by cephadm) 2024-03-21T13:15:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:28.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-21T13:15:28.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T13:15:28.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1163072488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:28.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/825369872' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:15:28.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:28.577 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]: dispatch 2024-03-21T13:15:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:29.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]': finished 2024-03-21T13:15:29.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: osdmap e144: 8 total, 8 up, 8 in 2024-03-21T13:15:29.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:29.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/904230982' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:29.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:29.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]': finished 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: osdmap e144: 8 total, 8 up, 8 in 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/904230982' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi089-79426-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi089-79628-1","app":"app1","key":"key1"}]': finished 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: osdmap e144: 8 total, 8 up, 8 in 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/904230982' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:29.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:29.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-21T13:15:29.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-21T13:15:29.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-21T13:15:29.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: seed 79628 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (1169 ms) 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (7 ms) 2024-03-21T13:15:29.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (5 ms) 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (5 ms) 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1 ms) 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (2 ms) 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1 ms) 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (6 ms) 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-21T13:15:29.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (1 ms) 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (3 ms) 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (5 ms) 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1685 ms) 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (379 ms) 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: done waiting, doing copies 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: done waiting 2024-03-21T13:15:29.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60887 ms) 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (5 ms) 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4076 ms) 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-21T13:15:29.613 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (1 ms) 2024-03-21T13:15:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: pgmap v182: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 KiB/s wr, 93 op/s 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:30.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: pgmap v182: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 KiB/s wr, 93 op/s 2024-03-21T13:15:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T13:15:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: pgmap v182: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 KiB/s wr, 93 op/s 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:30.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi089-79447-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/624198595' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]: dispatch 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:30.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (6 api_snapshots: Running main() from gmock_main.cc 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (6647 ms) 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4608 ms) 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (3137 ms) 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (3214 ms) 2024-03-21T13:15:30.609 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (17606 ms total) 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (6680 ms) 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4990 ms) 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6831 ms) 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (18501 ms total) 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-21T13:15:30.610 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3008 ms) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1984 ms) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2009 ms) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1999 ms) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9000 ms total) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4035 ms) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4003 ms) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8038 ms total) 2024-03-21T13:15:30.611 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: 2024-03-21T13:15:30.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-21T13:15:30.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (78327 ms total) 2024-03-21T13:15:30.612 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-21T13:15:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:31 smithi152 ceph-mon[34051]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T13:15:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2320041102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79426-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[30095]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2320041102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79426-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi089-79725-15"}]': finished 2024-03-21T13:15:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79628-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[39209]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T13:15:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2320041102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79426-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:32 smithi152 ceph-mon[34051]: pgmap v185: 356 pgs: 23 unknown, 31 creating+peering, 302 active+clean; 594 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2320041102' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79426-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:32 smithi152 ceph-mon[34051]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T13:15:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3507810904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi089-79447-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[30095]: pgmap v185: 356 pgs: 23 unknown, 31 creating+peering, 302 active+clean; 594 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2320041102' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79426-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[30095]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3507810904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi089-79447-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[39209]: pgmap v185: 356 pgs: 23 unknown, 31 creating+peering, 302 active+clean; 594 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2320041102' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79426-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[39209]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T13:15:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3507810904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi089-79447-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3507810904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi089-79447-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:33 smithi152 ceph-mon[34051]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T13:15:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2455054596' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3507810904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi089-79447-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[30095]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2455054596' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79628-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3507810904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi089-79447-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[39209]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T13:15:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2455054596' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:33.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: pgmap v188: 364 pgs: 72 unknown, 292 active+clean; 592 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:34.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:34.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:34.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T13:15:34.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.593 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.594 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2930414970' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.594 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.595 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: pgmap v188: 364 pgs: 72 unknown, 292 active+clean; 592 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:34.595 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:34.596 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:34.596 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T13:15:34.596 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.596 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.596 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2930414970' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.597 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.598 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: pgmap v188: 364 pgs: 72 unknown, 292 active+clean; 592 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:34.598 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:34.598 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79872-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:34.599 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T13:15:34.599 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.599 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.599 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2930414970' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:34.599 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.117 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:35.117 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:35.117 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:35.117 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T13:15:35.118 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.118 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.118 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/307200085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/307200085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi089-79426-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/307200085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:35.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:37.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:36 smithi152 ceph-mon[34051]: pgmap v191: 428 pgs: 136 unknown, 292 active+clean; 592 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:36 smithi152 ceph-mon[34051]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T13:15:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[30095]: pgmap v191: 428 pgs: 136 unknown, 292 active+clean; 592 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[30095]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T13:15:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:37.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[39209]: pgmap v191: 428 pgs: 136 unknown, 292 active+clean; 592 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:37.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79628-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:37.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi089-79447-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:37.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[39209]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T13:15:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:37.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:15:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:37.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:37.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T13:15:37.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/56487932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:37.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:37.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:37.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:37.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:37.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T13:15:37.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/56487932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:37.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:37.828 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.829 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.829 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.829 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:37.829 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:37.829 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:38.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:15:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T13:15:38.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/56487932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:38.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:38.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: pgmap v194: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/991983541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: pgmap v194: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/991983541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: pgmap v194: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi089-79426-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/991983541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:39.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:40.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:40.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T13:15:40.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1833681316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi089-79426-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1833681316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi089-79426-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi089-79447-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/155901024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:40.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T13:15:40.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]: dispatch 2024-03-21T13:15:40.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:40.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi089-79872-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:40.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79628-24"}]': finished 2024-03-21T13:15:40.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:40.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T13:15:40.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1833681316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi089-79426-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:41 smithi152 ceph-mon[34051]: pgmap v197: 324 pgs: 32 unknown, 292 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1833681316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi089-79426-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:41.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:41 smithi152 ceph-mon[34051]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T13:15:41.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:41.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/575440058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/978186420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79447-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[30095]: pgmap v197: 324 pgs: 32 unknown, 292 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1833681316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi089-79426-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[30095]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/575440058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/978186420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79447-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[39209]: pgmap v197: 324 pgs: 32 unknown, 292 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1833681316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi089-79426-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[39209]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T13:15:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/575440058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/978186420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79447-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:42 smithi152 ceph-mon[34051]: pgmap v200: 396 pgs: 10 creating+peering, 83 unknown, 303 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/978186420' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79447-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:42 smithi152 ceph-mon[34051]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T13:15:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:42 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[30095]: pgmap v200: 396 pgs: 10 creating+peering, 83 unknown, 303 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/978186420' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79447-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[30095]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[39209]: pgmap v200: 396 pgs: 10 creating+peering, 83 unknown, 303 active+clean; 592 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/978186420' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi089-79447-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:43.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi089-79628-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[39209]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T13:15:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:15:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:43.640 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: Running main() from gmock_main.cc 2024-03-21T13:15:43.640 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-21T13:15:43.640 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [----------] Global test environment set-up. 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (16681 ms) 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4387 ms) 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4162 ms) 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5835 ms) 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (10098 ms) 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (41163 ms total) 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: 2024-03-21T13:15:43.641 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5802 ms) 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (7006 ms) 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8008 ms) 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5998 ms) 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3996 ms) 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi089-80733-11. 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: Waiting for snaps to purge. 2024-03-21T13:15:43.642 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19063 ms) 2024-03-21T13:15:43.643 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (49873 ms total) 2024-03-21T13:15:43.643 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: 2024-03-21T13:15:43.643 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [----------] Global test environment tear-down 2024-03-21T13:15:43.643 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (91036 ms total) 2024-03-21T13:15:43.643 INFO:tasks.workunit.client.0.smithi089.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-21T13:15:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:43 smithi152 ceph-mon[34051]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T13:15:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3688137238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi089-79426-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:43 smithi089 ceph-mon[30095]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T13:15:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3688137238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi089-79426-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:43 smithi089 ceph-mon[39209]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T13:15:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3688137238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi089-79426-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:44.640 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-21T13:15:44.640 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-21T13:15:44.640 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: seed 79872 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (6646 ms) 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (4608 ms) 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3138 ms) 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (3212 ms) 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (6218 ms) 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (23822 ms total) 2024-03-21T13:15:44.641 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4905 ms) 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4902 ms) 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (8020 ms) 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3988 ms) 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2053 ms) 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-21T13:15:44.642 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2267-g960fbbe4/rpm/el9/BUILD/ceph-19.0.0-2267-g960fbbe4/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi089-79872-7 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17952 ms) 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (41820 ms total) 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (5 ms) 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4793 ms) 2024-03-21T13:15:44.643 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4798 ms total) 2024-03-21T13:15:44.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: 2024-03-21T13:15:44.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-21T13:15:44.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-21T13:15:44.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3006 ms) 2024-03-21T13:15:44.644 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-21T13:15:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: pgmap v203: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:15:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3688137238' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi089-79426-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T13:15:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3630701759' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi089-79628-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/813616722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: pgmap v203: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3688137238' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi089-79426-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3630701759' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi089-79628-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/813616722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: pgmap v203: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3688137238' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi089-79426-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3630701759' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi089-79628-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/813616722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3630701759' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi089-79628-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:45 smithi152 ceph-mon[34051]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T13:15:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3630701759' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi089-79628-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[39209]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3630701759' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi089-79628-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi089-79447-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[30095]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T13:15:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:47.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:46 smithi152 ceph-mon[34051]: pgmap v206: 332 pgs: 64 unknown, 268 active+clean; 592 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:47.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:46 smithi152 ceph-mon[34051]: osdmap e161: 8 total, 8 up, 8 in 2024-03-21T13:15:47.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2504172522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:47.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[30095]: pgmap v206: 332 pgs: 64 unknown, 268 active+clean; 592 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:47.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[30095]: osdmap e161: 8 total, 8 up, 8 in 2024-03-21T13:15:47.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2504172522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:47.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[39209]: pgmap v206: 332 pgs: 64 unknown, 268 active+clean; 592 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:47.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[39209]: osdmap e161: 8 total, 8 up, 8 in 2024-03-21T13:15:47.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2504172522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:47.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:15:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[30095]: osdmap e162: 8 total, 8 up, 8 in 2024-03-21T13:15:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2134877304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/160566576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[39209]: osdmap e162: 8 total, 8 up, 8 in 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2134877304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/160566576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:47.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:48.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:15:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi089-79426-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-mon[34051]: osdmap e162: 8 total, 8 up, 8 in 2024-03-21T13:15:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2134877304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/160566576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:48 smithi152 ceph-mon[34051]: pgmap v209: 364 pgs: 64 unknown, 32 creating+peering, 268 active+clean; 592 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:15:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:48 smithi152 ceph-mon[34051]: osdmap e163: 8 total, 8 up, 8 in 2024-03-21T13:15:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[30095]: pgmap v209: 364 pgs: 64 unknown, 32 creating+peering, 268 active+clean; 592 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[30095]: osdmap e163: 8 total, 8 up, 8 in 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[39209]: pgmap v209: 364 pgs: 64 unknown, 32 creating+peering, 268 active+clean; 592 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi089-79447-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi089-79628-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[39209]: osdmap e163: 8 total, 8 up, 8 in 2024-03-21T13:15:49.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: osdmap e164: 8 total, 8 up, 8 in 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2608702357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi089-79426-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2608702357' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi089-79426-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: osdmap e165: 8 total, 8 up, 8 in 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2053485747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: osdmap e164: 8 total, 8 up, 8 in 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2608702357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi089-79426-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2608702357' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi089-79426-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: osdmap e165: 8 total, 8 up, 8 in 2024-03-21T13:15:50.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2053485747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:15:50.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: osdmap e164: 8 total, 8 up, 8 in 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2608702357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi089-79426-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2608702357' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi089-79426-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:50.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: osdmap e165: 8 total, 8 up, 8 in 2024-03-21T13:15:50.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:50.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2053485747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:50.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:50.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:50.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:50 smithi152 ceph-mon[34051]: pgmap v212: 300 pgs: 32 unknown, 268 active+clean; 592 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:15:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:50 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:50 smithi089 ceph-mon[30095]: pgmap v212: 300 pgs: 32 unknown, 268 active+clean; 592 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:15:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:50 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:50 smithi089 ceph-mon[39209]: pgmap v212: 300 pgs: 32 unknown, 268 active+clean; 592 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:15:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:50 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: osdmap e166: 8 total, 8 up, 8 in 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi089-79426-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi089-79426-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: osdmap e167: 8 total, 8 up, 8 in 2024-03-21T13:15:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi089-79426-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: osdmap e166: 8 total, 8 up, 8 in 2024-03-21T13:15:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi089-79426-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi089-79426-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: osdmap e167: 8 total, 8 up, 8 in 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi089-79426-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi089-79447-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2459236761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: osdmap e166: 8 total, 8 up, 8 in 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]: dispatch 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi089-79426-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi089-79628-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi089-79872-16"}]': finished 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi089-79426-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: osdmap e167: 8 total, 8 up, 8 in 2024-03-21T13:15:52.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi089-79426-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: pgmap v215: 292 pgs: 9 creating+activating, 19 unknown, 264 active+clean; 592 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:15:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:53.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:53.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:53.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: osdmap e168: 8 total, 8 up, 8 in 2024-03-21T13:15:53.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2166269919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:53.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: pgmap v215: 292 pgs: 9 creating+activating, 19 unknown, 264 active+clean; 592 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: osdmap e168: 8 total, 8 up, 8 in 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2166269919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: pgmap v215: 292 pgs: 9 creating+activating, 19 unknown, 264 active+clean; 592 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: osdmap e168: 8 total, 8 up, 8 in 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:15:53.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2166269919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:53.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:54 smithi152 ceph-mon[34051]: pgmap v218: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:15:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi089-79426-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:55.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:55.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:54 smithi152 ceph-mon[34051]: osdmap e169: 8 total, 8 up, 8 in 2024-03-21T13:15:55.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:55.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:55.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[39209]: pgmap v218: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi089-79426-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[39209]: osdmap e169: 8 total, 8 up, 8 in 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[30095]: pgmap v218: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi089-79426-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi089-79447-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[30095]: osdmap e169: 8 total, 8 up, 8 in 2024-03-21T13:15:55.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:55.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:55.633 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosS8245 ms total) 2024-03-21T13:15:55.633 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: 2024-03-21T13:15:55.633 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-21T13:15:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-21T13:15:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (127 ms) 2024-03-21T13:15:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (127 ms total) 2024-03-21T13:15:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: 2024-03-21T13:15:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-21T13:15:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-21T13:15:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (39 ms) 2024-03-21T13:15:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-21T13:15:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (8 ms) 2024-03-21T13:15:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (47 ms total) 2024-03-21T13:15:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: 2024-03-21T13:15:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-21T13:15:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-21T13:15:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (51 ms) 2024-03-21T13:15:55.636 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-21T13:15:55.636 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (3 ms) 2024-03-21T13:15:55.636 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (54 ms total) 2024-03-21T13:15:55.636 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: 2024-03-21T13:15:55.636 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-21T13:15:55.636 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-21T13:15:55.637 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (45 ms) 2024-03-21T13:15:55.637 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-21T13:15:55.637 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (5 ms) 2024-03-21T13:15:55.637 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (50 ms total) 2024-03-21T13:15:55.637 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: 2024-03-21T13:15:55.637 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-21T13:15:55.637 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-21T13:15:55.638 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1014 ms) 2024-03-21T13:15:55.638 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1014 ms total) 2024-03-21T13:15:55.638 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: 2024-03-21T13:15:55.638 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-21T13:15:55.638 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (103389 ms total) 2024-03-21T13:15:55.638 INFO:tasks.workunit.client.0.smithi089.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-21T13:15:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:15:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:55 smithi152 ceph-mon[34051]: osdmap e170: 8 total, 8 up, 8 in 2024-03-21T13:15:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:56.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:56.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:15:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[30095]: osdmap e170: 8 total, 8 up, 8 in 2024-03-21T13:15:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:15:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[39209]: osdmap e170: 8 total, 8 up, 8 in 2024-03-21T13:15:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3528005583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]: dispatch 2024-03-21T13:15:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:56 smithi152 ceph-mon[34051]: pgmap v221: 276 pgs: 16 unknown, 260 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:56 smithi152 ceph-mon[34051]: osdmap e171: 8 total, 8 up, 8 in 2024-03-21T13:15:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1174395225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:57.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:57.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:57.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[30095]: pgmap v221: 276 pgs: 16 unknown, 260 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[30095]: osdmap e171: 8 total, 8 up, 8 in 2024-03-21T13:15:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1174395225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[39209]: pgmap v221: 276 pgs: 16 unknown, 260 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:15:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi089-79628-36"}]': finished 2024-03-21T13:15:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[39209]: osdmap e171: 8 total, 8 up, 8 in 2024-03-21T13:15:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1174395225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:57.620 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: Running main() from gmock_main.cc 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [----------] Global test environment set-up. 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.TooBig (9495 ms) 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (6846 ms) 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4619 ms) 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4720 ms) 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (4654 ms) 2024-03-21T13:15:57.621 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (5152 ms) 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (5108 ms) 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2865 ms) 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3953 ms) 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3039 ms) 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2964 ms) 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-21T13:15:57.622 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3025 ms) 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2982 ms) 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2989 ms) 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.Flush (3016 ms) 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2997 ms) 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3001 ms) 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3004 ms) 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-21T13:15:57.623 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3010 ms) 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3012 ms) 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2978 ms) 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3267 ms) 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2764 ms) 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3019 ms) 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3055 ms) 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-21T13:15:57.624 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3143 ms) 2024-03-21T13:15:57.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [----------] 26 tests from LibRadosAio (98677 ms total) 2024-03-21T13:15:57.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: 2024-03-21T13:15:57.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-21T13:15:57.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-21T13:15:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:15:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:15:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: osdmap e172: 8 total, 8 up, 8 in 2024-03-21T13:15:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[30095]: osdmap e173: 8 total, 8 up, 8 in 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: osdmap e172: 8 total, 8 up, 8 in 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:57 smithi089 ceph-mon[39209]: osdmap e173: 8 total, 8 up, 8 in 2024-03-21T13:15:58.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi089-79447-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: osdmap e172: 8 total, 8 up, 8 in 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]: dispatch 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1485198230' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi089-79426-27"}]': finished 2024-03-21T13:15:58.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:57 smithi152 ceph-mon[34051]: osdmap e173: 8 total, 8 up, 8 in 2024-03-21T13:15:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: pgmap v224: 300 pgs: 32 creating+peering, 268 active+clean; 597 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-21T13:15:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi089-79426-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi089-79426-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: osdmap e174: 8 total, 8 up, 8 in 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3335903885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi089-79426-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: pgmap v224: 300 pgs: 32 creating+peering, 268 active+clean; 597 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-21T13:15:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi089-79426-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi089-79426-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: osdmap e174: 8 total, 8 up, 8 in 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3335903885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi089-79426-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:59.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: pgmap v224: 300 pgs: 32 creating+peering, 268 active+clean; 597 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi089-79426-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi089-79426-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: osdmap e174: 8 total, 8 up, 8 in 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:15:59.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3335903885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:59.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi089-79426-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:15:59.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:15:59.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]: dispatch 2024-03-21T13:16:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:59 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:15:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:00.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:00.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:15:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi089-79447-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:01 smithi152 ceph-mon[34051]: pgmap v227: 300 pgs: 32 unknown, 268 active+clean; 597 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-21T13:16:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]': finished 2024-03-21T13:16:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:01 smithi152 ceph-mon[34051]: osdmap e175: 8 total, 8 up, 8 in 2024-03-21T13:16:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi089-79426-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:01.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:01 smithi152 ceph-mon[34051]: osdmap e176: 8 total, 8 up, 8 in 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[30095]: pgmap v227: 300 pgs: 32 unknown, 268 active+clean; 597 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]': finished 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[30095]: osdmap e175: 8 total, 8 up, 8 in 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi089-79426-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[30095]: osdmap e176: 8 total, 8 up, 8 in 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[39209]: pgmap v227: 300 pgs: 32 unknown, 268 active+clean; 597 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-13"}]': finished 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:01.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[39209]: osdmap e175: 8 total, 8 up, 8 in 2024-03-21T13:16:01.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi089-79426-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:01.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:01 smithi089 ceph-mon[39209]: osdmap e176: 8 total, 8 up, 8 in 2024-03-21T13:16:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: pgmap v230: 244 pgs: 3 creating+peering, 5 unknown, 236 active+clean; 587 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 0 op/s 2024-03-21T13:16:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: osdmap e177: 8 total, 8 up, 8 in 2024-03-21T13:16:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1449532903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: pgmap v230: 244 pgs: 3 creating+peering, 5 unknown, 236 active+clean; 587 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 0 op/s 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: osdmap e177: 8 total, 8 up, 8 in 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1449532903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: pgmap v230: 244 pgs: 3 creating+peering, 5 unknown, 236 active+clean; 587 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 0 op/s 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: osdmap e177: 8 total, 8 up, 8 in 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1449532903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:02.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (9487 ms) 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19336 ms) 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-21T13:16:03.601 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8803 ms) 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3984 ms) 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (4019 ms) 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2779 ms) 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3290 ms) 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2722 ms) 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2989 ms) 2024-03-21T13:16:03.602 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3007 ms) 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2997 ms) 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3256 ms) 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2746 ms) 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2997 ms) 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3024 ms) 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3003 ms) 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-21T13:16:03.603 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3027 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2972 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3013 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3011 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3035 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3025 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2901 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3019 ms) 2024-03-21T13:16:03.604 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-21T13:16:03.605 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3014 ms) 2024-03-21T13:16:03.605 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-21T13:16:03.605 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2971 ms) 2024-03-21T13:16:03.605 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-21T13:16:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:03 smithi152 ceph-mon[34051]: osdmap e178: 8 total, 8 up, 8 in 2024-03-21T13:16:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:16:04.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:04.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[30095]: osdmap e178: 8 total, 8 up, 8 in 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi089-79447-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[39209]: osdmap e178: 8 total, 8 up, 8 in 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:16:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: pgmap v233: 300 pgs: 64 unknown, 236 active+clean; 610 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-21T13:16:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: osdmap e179: 8 total, 8 up, 8 in 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: osdmap e180: 8 total, 8 up, 8 in 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3614042320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi089-79447-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: pgmap v233: 300 pgs: 64 unknown, 236 active+clean; 610 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: osdmap e179: 8 total, 8 up, 8 in 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: osdmap e180: 8 total, 8 up, 8 in 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3614042320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi089-79447-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: pgmap v233: 300 pgs: 64 unknown, 236 active+clean; 610 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: osdmap e179: 8 total, 8 up, 8 in 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]: dispatch 2024-03-21T13:16:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4104466982' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi089-79426-28"}]': finished 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: osdmap e180: 8 total, 8 up, 8 in 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3614042320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi089-79447-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3614042320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi089-79447-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]': finished 2024-03-21T13:16:06.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: osdmap e181: 8 total, 8 up, 8 in 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3614042320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi089-79447-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]': finished 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: osdmap e181: 8 total, 8 up, 8 in 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]: dispatch 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]: dispatch 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3614042320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi089-79447-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-15"}]': finished 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi089-79426-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: osdmap e181: 8 total, 8 up, 8 in 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]: dispatch 2024-03-21T13:16:06.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:06 smithi152 ceph-mon[34051]: pgmap v236: 300 pgs: 64 unknown, 236 active+clean; 610 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 8.0 KiB/s wr, 2 op/s 2024-03-21T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:06 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]': finished 2024-03-21T13:16:07.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[30095]: pgmap v236: 300 pgs: 64 unknown, 236 active+clean; 610 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 8.0 KiB/s wr, 2 op/s 2024-03-21T13:16:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]': finished 2024-03-21T13:16:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[39209]: pgmap v236: 300 pgs: 64 unknown, 236 active+clean; 610 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 8.0 KiB/s wr, 2 op/s 2024-03-21T13:16:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-15", "mode": "writeback"}]': finished 2024-03-21T13:16:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:16:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:08.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:16:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: pgmap v239: 268 pgs: 268 active+clean; 610 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-21T13:16:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-21T13:16:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: osdmap e182: 8 total, 8 up, 8 in 2024-03-21T13:16:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-21T13:16:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-21T13:16:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: osdmap e183: 8 total, 8 up, 8 in 2024-03-21T13:16:08.325 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1187326669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:16:08.326 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: pgmap v239: 268 pgs: 268 active+clean; 610 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: osdmap e182: 8 total, 8 up, 8 in 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: osdmap e183: 8 total, 8 up, 8 in 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1187326669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:08.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: pgmap v239: 268 pgs: 268 active+clean; 610 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-21T13:16:08.328 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: osdmap e182: 8 total, 8 up, 8 in 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi089-79426-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi089-79505-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: osdmap e183: 8 total, 8 up, 8 in 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1187326669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.329 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:08.330 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:16:08.330 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-21T13:16:09.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAionapshotsECPP.SnapRemovePP (2015 ms) 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2021 ms) 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2027 ms) 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9069 ms total) 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4021 ms) 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-21T13:16:09.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3967 ms) 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4237 ms) 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12225 ms total) 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (117316 ms total) 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-21T13:16:09.628 INFO:tasks.workunit.client.0.smithi089.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-21T13:16:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]': finished 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: osdmap e184: 8 total, 8 up, 8 in 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:16:10.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]': finished 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: osdmap e184: 8 total, 8 up, 8 in 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:16:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi089-79447-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi089-79505-1","var":"pgp_num","val":"11"}]': finished 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: osdmap e184: 8 total, 8 up, 8 in 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/742198243' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]: dispatch 2024-03-21T13:16:10.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]: dispatch 2024-03-21T13:16:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: pgmap v242: 300 pgs: 40 unknown, 260 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]': finished 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: osdmap e185: 8 total, 8 up, 8 in 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: pgmap v242: 300 pgs: 40 unknown, 260 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]': finished 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: osdmap e185: 8 total, 8 up, 8 in 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: pgmap v242: 300 pgs: 40 unknown, 260 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1586496537' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi089-79872-21"}]': finished 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-15"}]': finished 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: osdmap e185: 8 total, 8 up, 8 in 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:11.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:11.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: osdmap e186: 8 total, 8 up, 8 in 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/344220148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: pgmap v245: 228 pgs: 11 creating+peering, 21 unknown, 196 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: osdmap e187: 8 total, 8 up, 8 in 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:12.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: osdmap e186: 8 total, 8 up, 8 in 2024-03-21T13:16:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/344220148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: pgmap v245: 228 pgs: 11 creating+peering, 21 unknown, 196 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: osdmap e187: 8 total, 8 up, 8 in 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi089-79505-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: osdmap e186: 8 total, 8 up, 8 in 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2483604254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/344220148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: pgmap v245: 228 pgs: 11 creating+peering, 21 unknown, 196 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.327 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi089-79426-29"}]': finished 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi089-79447-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: osdmap e187: 8 total, 8 up, 8 in 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:12.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: osdmap e188: 8 total, 8 up, 8 in 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:13.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: osdmap e188: 8 total, 8 up, 8 in 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi089-79505-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi089-79426-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: osdmap e188: 8 total, 8 up, 8 in 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:13.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:13.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout:ntry=4 expected=4 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:c4fdafeb:::6:head expected=10:c4fdafeb:::6:head 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=6 expected=6 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:02547ec2:::1:head expected=10:02547ec2:::1:head 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=1 expected=1 2024-03-21T13:16:13.684 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:863748b0:::15:head 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:863748b0:::15:head expected=10:863748b0:::15:head 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=15 expected=15 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:6cac518f:::0:head expected=10:6cac518f:::0:head 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=0 expected=0 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:62a1935d:::14:head expected=10:62a1935d:::14:head 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=14 expected=14 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:566253c9:::13:head 2024-03-21T13:16:13.685 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:566253c9:::13:head expected=10:566253c9:::13:head 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=13 expected=13 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:5c6b0b28:::7:head expected=10:5c6b0b28:::7:head 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=7 expected=7 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : cursor()=10:f905c69b:::2:head expected=10:f905c69b:::2:head 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: : entry=2 expected=2 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (189 ms) 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (61785 ms) 2024-03-21T13:16:13.686 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: split 0/5 -> MIN 10:33333333::::head 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: split 1/5 -> 10:33333333::::head 10:66666666::::head 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: split 2/5 -> 10:66666666::::head 10:99999999::::head 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: split 3/5 -> 10:99999999::::head 10:cccccccc::::head 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: split 4/5 -> 10:cccccccc::::head MAX 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (45002 ms) 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] 7 tests from LibRadosList (107314 ms total) 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1034 ms) 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset foo1,foo2,foo3 2024-03-21T13:16:13.687 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo1 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo2 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo3 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset foo1,foo4,foo5 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo4 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo5 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo1 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset foo6,foo7 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo7 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: foo6 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns1:foo4 2024-03-21T13:16:13.688 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns1:foo5 2024-03-21T13:16:13.689 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns2:foo7 2024-03-21T13:16:13.689 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns2:foo6 2024-03-21T13:16:13.689 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: ns1:foo1 2024-03-21T13:16:13.689 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: :foo1 2024-03-21T13:16:13.689 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: :foo2 2024-03-21T13:16:13.689 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: :foo3 2024-03-21T13:16:13.689 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (33 ms) 2024-03-21T13:16:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: pgmap v248: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: osdmap e189: 8 total, 8 up, 8 in 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1046145648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]': finished 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]: dispatch 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: osdmap e190: 8 total, 8 up, 8 in 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]: dispatch 2024-03-21T13:16:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: pgmap v248: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: osdmap e189: 8 total, 8 up, 8 in 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1046145648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]': finished 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]: dispatch 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: osdmap e190: 8 total, 8 up, 8 in 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]: dispatch 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: pgmap v248: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: osdmap e189: 8 total, 8 up, 8 in 2024-03-21T13:16:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1046145648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi089-79426-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-17"}]': finished 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi089-79447-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]: dispatch 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: osdmap e190: 8 total, 8 up, 8 in 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]: dispatch 2024-03-21T13:16:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:16.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1046145648' entity='client.admin' cmd=[{ 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "prefix": "osd pool set", 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "var": "eio", 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "val": "true" 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: }]: dispatch 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{ 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "prefix": "osd pool set", 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "var": "eio", 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "val": "true" 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: }]: dispatch 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]': finished 2024-03-21T13:16:16.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{ 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "prefix": "osd pool set", 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "var": "eio", 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: "val": "true" 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: }]': finished 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: osdmap e191: 8 total, 8 up, 8 in 2024-03-21T13:16:16.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1046145648' entity='client.admin' cmd=[{ 2024-03-21T13:16:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "prefix": "osd pool set", 2024-03-21T13:16:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "var": "eio", 2024-03-21T13:16:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "val": "true" 2024-03-21T13:16:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: }]: dispatch 2024-03-21T13:16:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{ 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "prefix": "osd pool set", 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "var": "eio", 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "val": "true" 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: }]: dispatch 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]': finished 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{ 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "prefix": "osd pool set", 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "var": "eio", 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: "val": "true" 2024-03-21T13:16:16.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: }]': finished 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: osdmap e191: 8 total, 8 up, 8 in 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1046145648' entity='client.admin' cmd=[{ 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "prefix": "osd pool set", 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "var": "eio", 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "val": "true" 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: }]: dispatch 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{ 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "prefix": "osd pool set", 2024-03-21T13:16:16.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "var": "eio", 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "val": "true" 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: }]: dispatch 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-17", "mode": "writeback"}]': finished 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{ 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "prefix": "osd pool set", 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "pool": "PoolEIOFlag_smithi089-79447-33", 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "var": "eio", 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: "val": "true" 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: }]': finished 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3029639012' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:16.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: osdmap e191: 8 total, 8 up, 8 in 2024-03-21T13:16:16.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]: dispatch 2024-03-21T13:16:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: pgmap v251: 268 pgs: 72 unknown, 196 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-21T13:16:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: osdmap e192: 8 total, 8 up, 8 in 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: pgmap v251: 268 pgs: 72 unknown, 196 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: osdmap e192: 8 total, 8 up, 8 in 2024-03-21T13:16:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:17.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:17.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:17.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: pgmap v251: 268 pgs: 72 unknown, 196 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-21T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi089-79505-2"}]': finished 2024-03-21T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:17.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: osdmap e192: 8 total, 8 up, 8 in 2024-03-21T13:16:17.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:17.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]: dispatch 2024-03-21T13:16:17.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:17.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:17.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:16:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:16:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:18.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:16:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:16:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: pgmap v254: 228 pgs: 228 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:16:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]': finished 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: osdmap e193: 8 total, 8 up, 8 in 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1187553524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: pgmap v254: 228 pgs: 228 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:16:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]': finished 2024-03-21T13:16:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: osdmap e193: 8 total, 8 up, 8 in 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1187553524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: pgmap v254: 228 pgs: 228 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-17"}]': finished 2024-03-21T13:16:18.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:18.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: osdmap e193: 8 total, 8 up, 8 in 2024-03-21T13:16:18.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/917473673' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:18.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]: dispatch 2024-03-21T13:16:18.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1187553524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:18.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:19.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-21T13:16:19.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 1 0 2024-03-21T13:16:19.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 10 0 2024-03-21T13:16:19.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 13 0 2024-03-21T13:16:19.625 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 7 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 14 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 0 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 15 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 11 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 5 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 8 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 6 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 3 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 4 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 12 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 9 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 2 0 2024-03-21T13:16:19.626 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (43 ms) 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] 3 tests from LibRadosListEC (1111 ms total) 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3026 ms) 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] 1 test from LibRadosListNP (3026 ms total) 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [----------] Global test environment tear-down 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [==========] 11 tests from 3 test suites ran. (127409 ms total) 2024-03-21T13:16:19.627 INFO:tasks.workunit.client.0.smithi089.stdout: api_list: [ PASSED ] 11 tests. 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79505-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: osdmap e194: 8 total, 8 up, 8 in 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi089-79505-3","pool2":"test-rados-api-smithi089-79505-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:20.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79505-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: osdmap e194: 8 total, 8 up, 8 in 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi089-79505-3","pool2":"test-rados-api-smithi089-79505-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79505-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi089-79426-30"}]': finished 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi089-79447-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: osdmap e194: 8 total, 8 up, 8 in 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi089-79505-3","pool2":"test-rados-api-smithi089-79505-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: pgmap v257: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi089-79505-3","pool2":"test-rados-api-smithi089-79505-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-21T13:16:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: osdmap e195: 8 total, 8 up, 8 in 2024-03-21T13:16:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:21.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: pgmap v257: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi089-79505-3","pool2":"test-rados-api-smithi089-79505-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: osdmap e195: 8 total, 8 up, 8 in 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: pgmap v257: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:21.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2275307910' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi089-79505-3","pool2":"test-rados-api-smithi089-79505-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-21T13:16:21.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi089-79426-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:21.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: osdmap e195: 8 total, 8 up, 8 in 2024-03-21T13:16:21.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:21.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:21.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:21.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: pgmap v260: 260 pgs: 11 creating+peering, 42 unknown, 207 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: osdmap e196: 8 total, 8 up, 8 in 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/47977353' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: osdmap e197: 8 total, 8 up, 8 in 2024-03-21T13:16:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: pgmap v260: 260 pgs: 11 creating+peering, 42 unknown, 207 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: osdmap e196: 8 total, 8 up, 8 in 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/47977353' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: osdmap e197: 8 total, 8 up, 8 in 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: pgmap v260: 260 pgs: 11 creating+peering, 42 unknown, 207 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: osdmap e196: 8 total, 8 up, 8 in 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/47977353' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi089-79426-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:22.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi089-79447-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:22.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:22.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: osdmap e197: 8 total, 8 up, 8 in 2024-03-21T13:16:22.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]': finished 2024-03-21T13:16:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:23 smithi152 ceph-mon[34051]: osdmap e198: 8 total, 8 up, 8 in 2024-03-21T13:16:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]: dispatch 2024-03-21T13:16:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]: dispatch 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]': finished 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[30095]: osdmap e198: 8 total, 8 up, 8 in 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]: dispatch 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]: dispatch 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-19"}]': finished 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[39209]: osdmap e198: 8 total, 8 up, 8 in 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]: dispatch 2024-03-21T13:16:24.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]: dispatch 2024-03-21T13:16:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: pgmap v263: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]': finished 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: osdmap e199: 8 total, 8 up, 8 in 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2053201347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:25.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:24 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: pgmap v263: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]': finished 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: osdmap e199: 8 total, 8 up, 8 in 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2053201347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: pgmap v263: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-19", "mode": "writeback"}]': finished 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: osdmap e199: 8 total, 8 up, 8 in 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2053201347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:25.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:25.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:24 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2053201347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: osdmap e200: 8 total, 8 up, 8 in 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:25.973 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2053201347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: osdmap e200: 8 total, 8 up, 8 in 2024-03-21T13:16:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2053201347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: osdmap e200: 8 total, 8 up, 8 in 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3070988226' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]: dispatch 2024-03-21T13:16:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]: dispatch 2024-03-21T13:16:27.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: pgmap v266: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:27.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:27.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]': finished 2024-03-21T13:16:27.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:27.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: osdmap e201: 8 total, 8 up, 8 in 2024-03-21T13:16:27.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/251829455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:27.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: pgmap v266: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]': finished 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: osdmap e201: 8 total, 8 up, 8 in 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/251829455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: pgmap v266: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-19"}]': finished 2024-03-21T13:16:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi089-79426-31"}]': finished 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: osdmap e201: 8 total, 8 up, 8 in 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/251829455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:27.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:16:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:27.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/251829455' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:27.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:27.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[39209]: osdmap e202: 8 total, 8 up, 8 in 2024-03-21T13:16:27.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/251829455' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[30095]: osdmap e202: 8 total, 8 up, 8 in 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:16:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:28.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:16:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/251829455' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi089-79426-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-mon[34051]: osdmap e202: 8 total, 8 up, 8 in 2024-03-21T13:16:28.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:28.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:28.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:16:28.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: pgmap v269: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:16:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:16:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: osdmap e203: 8 total, 8 up, 8 in 2024-03-21T13:16:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3345555376' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: pgmap v269: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: osdmap e203: 8 total, 8 up, 8 in 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3345555376' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: pgmap v269: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: osdmap e203: 8 total, 8 up, 8 in 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3345555376' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: osdmap e204: 8 total, 8 up, 8 in 2024-03-21T13:16:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:29 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: osdmap e204: 8 total, 8 up, 8 in 2024-03-21T13:16:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:30.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:30.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi089-79426-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:30.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi089-79447-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: osdmap e204: 8 total, 8 up, 8 in 2024-03-21T13:16:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:29 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:31.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:30 smithi152 ceph-mon[34051]: pgmap v272: 332 pgs: 72 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:31.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:30 smithi152 ceph-mon[34051]: osdmap e205: 8 total, 8 up, 8 in 2024-03-21T13:16:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:31.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[30095]: pgmap v272: 332 pgs: 72 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[30095]: osdmap e205: 8 total, 8 up, 8 in 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[39209]: pgmap v272: 332 pgs: 72 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[39209]: osdmap e205: 8 total, 8 up, 8 in 2024-03-21T13:16:31.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]': finished 2024-03-21T13:16:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:31 smithi152 ceph-mon[34051]: osdmap e206: 8 total, 8 up, 8 in 2024-03-21T13:16:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]: dispatch 2024-03-21T13:16:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]: dispatch 2024-03-21T13:16:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:32.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:31 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]': finished 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[30095]: osdmap e206: 8 total, 8 up, 8 in 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]: dispatch 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]: dispatch 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:32.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-21"}]': finished 2024-03-21T13:16:32.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[39209]: osdmap e206: 8 total, 8 up, 8 in 2024-03-21T13:16:32.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]: dispatch 2024-03-21T13:16:32.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]: dispatch 2024-03-21T13:16:32.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:32.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:32.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:31 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:33.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: pgmap v275: 260 pgs: 24 unknown, 236 active+clean; 584 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]': finished 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: osdmap e207: 8 total, 8 up, 8 in 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: osdmap e208: 8 total, 8 up, 8 in 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/947429808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: pgmap v275: 260 pgs: 24 unknown, 236 active+clean; 584 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]': finished 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: osdmap e207: 8 total, 8 up, 8 in 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: osdmap e208: 8 total, 8 up, 8 in 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/947429808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: pgmap v275: 260 pgs: 24 unknown, 236 active+clean; 584 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-21", "mode": "writeback"}]': finished 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: osdmap e207: 8 total, 8 up, 8 in 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1674080733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]: dispatch 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi089-79426-32"}]': finished 2024-03-21T13:16:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:33.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:33.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: osdmap e208: 8 total, 8 up, 8 in 2024-03-21T13:16:33.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]: dispatch 2024-03-21T13:16:33.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/947429808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:33.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:34.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]': finished 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: osdmap e209: 8 total, 8 up, 8 in 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]': finished 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: osdmap e209: 8 total, 8 up, 8 in 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:34.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-21"}]': finished 2024-03-21T13:16:34.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:34.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi089-79426-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:34.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: osdmap e209: 8 total, 8 up, 8 in 2024-03-21T13:16:34.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:34.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:35.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:34 smithi152 ceph-mon[34051]: pgmap v278: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:35.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:34 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:35.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:34 smithi089 ceph-mon[30095]: pgmap v278: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:35.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:34 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:35.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:34 smithi089 ceph-mon[39209]: pgmap v278: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:35.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:34 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:35 smithi152 ceph-mon[34051]: osdmap e210: 8 total, 8 up, 8 in 2024-03-21T13:16:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1382334545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:36.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1382334545' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:36.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:35 smithi152 ceph-mon[34051]: osdmap e211: 8 total, 8 up, 8 in 2024-03-21T13:16:36.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[39209]: osdmap e210: 8 total, 8 up, 8 in 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1382334545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1382334545' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[39209]: osdmap e211: 8 total, 8 up, 8 in 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[30095]: osdmap e210: 8 total, 8 up, 8 in 2024-03-21T13:16:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1382334545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi089-79426-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:36.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1382334545' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:36.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[30095]: osdmap e211: 8 total, 8 up, 8 in 2024-03-21T13:16:36.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:36.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:36 smithi152 ceph-mon[34051]: pgmap v281: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:37.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:36 smithi152 ceph-mon[34051]: osdmap e212: 8 total, 8 up, 8 in 2024-03-21T13:16:37.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2450241014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[30095]: pgmap v281: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[30095]: osdmap e212: 8 total, 8 up, 8 in 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2450241014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[39209]: pgmap v281: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[39209]: osdmap e212: 8 total, 8 up, 8 in 2024-03-21T13:16:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2450241014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:16:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:16:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:38.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:16:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:16:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:38 smithi152 ceph-mon[34051]: pgmap v284: 332 pgs: 32 unknown, 40 creating+peering, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2450241014' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:38 smithi152 ceph-mon[34051]: osdmap e213: 8 total, 8 up, 8 in 2024-03-21T13:16:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[30095]: pgmap v284: 332 pgs: 32 unknown, 40 creating+peering, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2450241014' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[30095]: osdmap e213: 8 total, 8 up, 8 in 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[39209]: pgmap v284: 332 pgs: 32 unknown, 40 creating+peering, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:39.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2450241014' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[39209]: osdmap e213: 8 total, 8 up, 8 in 2024-03-21T13:16:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:40.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:40.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:40.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: osdmap e214: 8 total, 8 up, 8 in 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2559067317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]': finished 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: osdmap e214: 8 total, 8 up, 8 in 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2559067317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]': finished 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: osdmap e214: 8 total, 8 up, 8 in 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:40.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2559067317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:40.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4289623359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:40.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:40.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]: dispatch 2024-03-21T13:16:40.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:40.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-23"}]': finished 2024-03-21T13:16:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: pgmap v287: 356 pgs: 64 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]: dispatch 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: osdmap e215: 8 total, 8 up, 8 in 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]: dispatch 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:41.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:41.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:41.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]': finished 2024-03-21T13:16:41.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:41.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: osdmap e216: 8 total, 8 up, 8 in 2024-03-21T13:16:41.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/109974534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:41.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: pgmap v287: 356 pgs: 64 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:41.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:41.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:41.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]: dispatch 2024-03-21T13:16:41.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: osdmap e215: 8 total, 8 up, 8 in 2024-03-21T13:16:41.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]: dispatch 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]': finished 2024-03-21T13:16:41.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:41.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: osdmap e216: 8 total, 8 up, 8 in 2024-03-21T13:16:41.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/109974534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:41.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: pgmap v287: 356 pgs: 64 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:41.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:41.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi089-79426-33"}]': finished 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]: dispatch 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: osdmap e215: 8 total, 8 up, 8 in 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]: dispatch 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:41.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:41.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:41.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-23", "mode": "writeback"}]': finished 2024-03-21T13:16:41.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi089-79426-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:41.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: osdmap e216: 8 total, 8 up, 8 in 2024-03-21T13:16:41.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/109974534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:41.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:41.079 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:42 smithi152 ceph-mon[34051]: pgmap v290: 388 pgs: 72 unknown, 32 creating+peering, 284 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:16:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/109974534' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:42 smithi152 ceph-mon[34051]: osdmap e217: 8 total, 8 up, 8 in 2024-03-21T13:16:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:42 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[30095]: pgmap v290: 388 pgs: 72 unknown, 32 creating+peering, 284 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/109974534' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[30095]: osdmap e217: 8 total, 8 up, 8 in 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[39209]: pgmap v290: 388 pgs: 72 unknown, 32 creating+peering, 284 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/109974534' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi089-79447-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:43.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[39209]: osdmap e217: 8 total, 8 up, 8 in 2024-03-21T13:16:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:43 smithi152 ceph-mon[34051]: osdmap e218: 8 total, 8 up, 8 in 2024-03-21T13:16:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:44.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:43 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[30095]: osdmap e218: 8 total, 8 up, 8 in 2024-03-21T13:16:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi089-79426-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:44.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:44.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:44.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[39209]: osdmap e218: 8 total, 8 up, 8 in 2024-03-21T13:16:44.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]: dispatch 2024-03-21T13:16:44.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:43 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:44 smithi152 ceph-mon[34051]: pgmap v293: 364 pgs: 8 unknown, 356 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-21T13:16:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]': finished 2024-03-21T13:16:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:44 smithi152 ceph-mon[34051]: osdmap e219: 8 total, 8 up, 8 in 2024-03-21T13:16:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:44 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:44 smithi152 ceph-mon[34051]: osdmap e220: 8 total, 8 up, 8 in 2024-03-21T13:16:45.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[30095]: pgmap v293: 364 pgs: 8 unknown, 356 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]': finished 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[30095]: osdmap e219: 8 total, 8 up, 8 in 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[30095]: osdmap e220: 8 total, 8 up, 8 in 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[39209]: pgmap v293: 364 pgs: 8 unknown, 356 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-23"}]': finished 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[39209]: osdmap e219: 8 total, 8 up, 8 in 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:45.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:44 smithi089 ceph-mon[39209]: osdmap e220: 8 total, 8 up, 8 in 2024-03-21T13:16:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: pgmap v296: 260 pgs: 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 2 op/s 2024-03-21T13:16:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: osdmap e221: 8 total, 8 up, 8 in 2024-03-21T13:16:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: pgmap v296: 260 pgs: 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 2 op/s 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: osdmap e221: 8 total, 8 up, 8 in 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: pgmap v296: 260 pgs: 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 2 op/s 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: osdmap e221: 8 total, 8 up, 8 in 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/247071787' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]: dispatch 2024-03-21T13:16:46.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:47.747 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:16:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:48.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:16:48.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: osdmap e222: 8 total, 8 up, 8 in 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:48.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: osdmap e222: 8 total, 8 up, 8 in 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:48.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi089-79426-34"}]': finished 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: osdmap e222: 8 total, 8 up, 8 in 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:48.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: pgmap v299: 228 pgs: 32 creating+peering, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: osdmap e223: 8 total, 8 up, 8 in 2024-03-21T13:16:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2279326265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:49.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:49.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: pgmap v299: 228 pgs: 32 creating+peering, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: osdmap e223: 8 total, 8 up, 8 in 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2279326265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:49.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: pgmap v299: 228 pgs: 32 creating+peering, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi089-79426-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: osdmap e223: 8 total, 8 up, 8 in 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2279326265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:49.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:50.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:50.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: osdmap e224: 8 total, 8 up, 8 in 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]': finished 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]: dispatch 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: osdmap e225: 8 total, 8 up, 8 in 2024-03-21T13:16:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]: dispatch 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: osdmap e224: 8 total, 8 up, 8 in 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]': finished 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]: dispatch 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: osdmap e225: 8 total, 8 up, 8 in 2024-03-21T13:16:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]: dispatch 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi089-79447-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: osdmap e224: 8 total, 8 up, 8 in 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi089-79426-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-25"}]': finished 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]: dispatch 2024-03-21T13:16:50.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: osdmap e225: 8 total, 8 up, 8 in 2024-03-21T13:16:50.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]: dispatch 2024-03-21T13:16:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:50 smithi152 ceph-mon[34051]: pgmap v302: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:50 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]': finished 2024-03-21T13:16:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:50 smithi152 ceph-mon[34051]: osdmap e226: 8 total, 8 up, 8 in 2024-03-21T13:16:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/633972895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi089-79447-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:51.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[30095]: pgmap v302: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]': finished 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[30095]: osdmap e226: 8 total, 8 up, 8 in 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/633972895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi089-79447-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[39209]: pgmap v302: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-25", "mode": "writeback"}]': finished 2024-03-21T13:16:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[39209]: osdmap e226: 8 total, 8 up, 8 in 2024-03-21T13:16:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/633972895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi089-79447-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:16:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: pgmap v305: 268 pgs: 30 unknown, 28 creating+peering, 210 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:16:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/633972895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi089-79447-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: osdmap e227: 8 total, 8 up, 8 in 2024-03-21T13:16:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: pgmap v305: 268 pgs: 30 unknown, 28 creating+peering, 210 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/633972895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi089-79447-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: osdmap e227: 8 total, 8 up, 8 in 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:53.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: pgmap v305: 268 pgs: 30 unknown, 28 creating+peering, 210 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:16:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/633972895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi089-79447-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:16:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: osdmap e227: 8 total, 8 up, 8 in 2024-03-21T13:16:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]: dispatch 2024-03-21T13:16:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]': finished 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: osdmap e228: 8 total, 8 up, 8 in 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:54.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:53 smithi152 ceph-mon[34051]: osdmap e229: 8 total, 8 up, 8 in 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]': finished 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: osdmap e228: 8 total, 8 up, 8 in 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[30095]: osdmap e229: 8 total, 8 up, 8 in 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-25"}]': finished 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:54.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: osdmap e228: 8 total, 8 up, 8 in 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2942428781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi089-79426-35"}]': finished 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:54.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:53 smithi089 ceph-mon[39209]: osdmap e229: 8 total, 8 up, 8 in 2024-03-21T13:16:55.222 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: pgmap v308: 228 pgs: 228 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:55.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: pgmap v308: 228 pgs: 228 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:55.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: pgmap v308: 228 pgs: 228 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:16:55.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi089-79426-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:16:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:55 smithi152 ceph-mon[34051]: osdmap e230: 8 total, 8 up, 8 in 2024-03-21T13:16:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[39209]: osdmap e230: 8 total, 8 up, 8 in 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[30095]: osdmap e230: 8 total, 8 up, 8 in 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:16:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:16:56.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:16:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: pgmap v311: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T13:16:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: osdmap e231: 8 total, 8 up, 8 in 2024-03-21T13:16:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:57.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:16:57.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: osdmap e232: 8 total, 8 up, 8 in 2024-03-21T13:16:57.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:57.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:57.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: pgmap v311: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T13:16:57.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:57.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: osdmap e231: 8 total, 8 up, 8 in 2024-03-21T13:16:57.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:57.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:16:57.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: osdmap e232: 8 total, 8 up, 8 in 2024-03-21T13:16:57.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: pgmap v311: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: osdmap e231: 8 total, 8 up, 8 in 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi089-79426-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: osdmap e232: 8 total, 8 up, 8 in 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:57.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:16:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:16:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:16:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:16:58.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:16:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:16:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: pgmap v314: 244 pgs: 7 creating+peering, 9 unknown, 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-21T13:16:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: osdmap e233: 8 total, 8 up, 8 in 2024-03-21T13:16:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:16:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: pgmap v314: 244 pgs: 7 creating+peering, 9 unknown, 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: osdmap e233: 8 total, 8 up, 8 in 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: pgmap v314: 244 pgs: 7 creating+peering, 9 unknown, 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: osdmap e233: 8 total, 8 up, 8 in 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:16:58.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:17:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]': finished 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]: dispatch 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: osdmap e234: 8 total, 8 up, 8 in 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]: dispatch 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:16:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]': finished 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: osdmap e234: 8 total, 8 up, 8 in 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:00.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-27"}]': finished 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]: dispatch 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: osdmap e234: 8 total, 8 up, 8 in 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2262743389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]: dispatch 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]: dispatch 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:16:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: pgmap v317: 228 pgs: 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:17:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]': finished 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: osdmap e235: 8 total, 8 up, 8 in 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: pgmap v317: 228 pgs: 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]': finished 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: osdmap e235: 8 total, 8 up, 8 in 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: pgmap v317: 228 pgs: 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-27", "mode": "writeback"}]': finished 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-46"}]': finished 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: osdmap e235: 8 total, 8 up, 8 in 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/127210913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:01.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]: dispatch 2024-03-21T13:17:01.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:01.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:01.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: osdmap e236: 8 total, 8 up, 8 in 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:02.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:02.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:02.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: osdmap e236: 8 total, 8 up, 8 in 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi089-79426-36"}]': finished 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi089-79447-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: osdmap e236: 8 total, 8 up, 8 in 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:02.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:02.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: pgmap v320: 228 pgs: 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:17:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:17:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: osdmap e237: 8 total, 8 up, 8 in 2024-03-21T13:17:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:03.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: pgmap v320: 228 pgs: 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: osdmap e237: 8 total, 8 up, 8 in 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: pgmap v320: 228 pgs: 228 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:17:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: osdmap e237: 8 total, 8 up, 8 in 2024-03-21T13:17:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]: dispatch 2024-03-21T13:17:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:03 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]': finished 2024-03-21T13:17:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:03 smithi152 ceph-mon[34051]: osdmap e238: 8 total, 8 up, 8 in 2024-03-21T13:17:04.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]': finished 2024-03-21T13:17:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[30095]: osdmap e238: 8 total, 8 up, 8 in 2024-03-21T13:17:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi089-79447-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-27"}]': finished 2024-03-21T13:17:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:03 smithi089 ceph-mon[39209]: osdmap e238: 8 total, 8 up, 8 in 2024-03-21T13:17:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: pgmap v323: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:17:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: osdmap e239: 8 total, 8 up, 8 in 2024-03-21T13:17:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: osdmap e240: 8 total, 8 up, 8 in 2024-03-21T13:17:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: pgmap v323: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: osdmap e239: 8 total, 8 up, 8 in 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: osdmap e240: 8 total, 8 up, 8 in 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: pgmap v323: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:17:05.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi089-79426-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: osdmap e239: 8 total, 8 up, 8 in 2024-03-21T13:17:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: osdmap e240: 8 total, 8 up, 8 in 2024-03-21T13:17:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: pgmap v326: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-21T13:17:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: osdmap e241: 8 total, 8 up, 8 in 2024-03-21T13:17:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: pgmap v326: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-21T13:17:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: osdmap e241: 8 total, 8 up, 8 in 2024-03-21T13:17:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: pgmap v326: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: osdmap e241: 8 total, 8 up, 8 in 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4114910611' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]: dispatch 2024-03-21T13:17:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:17:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: osdmap e242: 8 total, 8 up, 8 in 2024-03-21T13:17:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: osdmap e242: 8 total, 8 up, 8 in 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:07.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:07.911 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi089-79447-47"}]': finished 2024-03-21T13:17:07.911 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:07.911 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: osdmap e242: 8 total, 8 up, 8 in 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2012417413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]: dispatch 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:07.912 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:08.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:17:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:17:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: pgmap v329: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:17:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: osdmap e243: 8 total, 8 up, 8 in 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:09.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi089-79426-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: pgmap v329: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: osdmap e243: 8 total, 8 up, 8 in 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi089-79426-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: pgmap v329: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi089-79426-37"}]': finished 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi089-79447-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: osdmap e243: 8 total, 8 up, 8 in 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:09.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:09.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:09.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:09.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi089-79426-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi089-79426-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: osdmap e244: 8 total, 8 up, 8 in 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi089-79426-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:10.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi089-79426-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: osdmap e244: 8 total, 8 up, 8 in 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi089-79426-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi089-79426-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: osdmap e244: 8 total, 8 up, 8 in 2024-03-21T13:17:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi089-79426-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:10.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:10.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi089-79447-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: pgmap v332: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-21T13:17:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]': finished 2024-03-21T13:17:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]: dispatch 2024-03-21T13:17:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: osdmap e245: 8 total, 8 up, 8 in 2024-03-21T13:17:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]: dispatch 2024-03-21T13:17:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi089-79426-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]': finished 2024-03-21T13:17:11.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:10 smithi152 ceph-mon[34051]: osdmap e246: 8 total, 8 up, 8 in 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: pgmap v332: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]': finished 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]: dispatch 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: osdmap e245: 8 total, 8 up, 8 in 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]: dispatch 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi089-79426-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]': finished 2024-03-21T13:17:11.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[30095]: osdmap e246: 8 total, 8 up, 8 in 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: pgmap v332: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-29"}]': finished 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]: dispatch 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: osdmap e245: 8 total, 8 up, 8 in 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]: dispatch 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi089-79426-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-29", "mode": "writeback"}]': finished 2024-03-21T13:17:11.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:10 smithi089 ceph-mon[39209]: osdmap e246: 8 total, 8 up, 8 in 2024-03-21T13:17:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:12 smithi152 ceph-mon[34051]: pgmap v335: 244 pgs: 3 creating+peering, 10 unknown, 231 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:12 smithi152 ceph-mon[34051]: osdmap e247: 8 total, 8 up, 8 in 2024-03-21T13:17:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:12 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:12 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[39209]: pgmap v335: 244 pgs: 3 creating+peering, 10 unknown, 231 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[39209]: osdmap e247: 8 total, 8 up, 8 in 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[30095]: pgmap v335: 244 pgs: 3 creating+peering, 10 unknown, 231 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[30095]: osdmap e247: 8 total, 8 up, 8 in 2024-03-21T13:17:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:12 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: osdmap e248: 8 total, 8 up, 8 in 2024-03-21T13:17:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: osdmap e249: 8 total, 8 up, 8 in 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: osdmap e248: 8 total, 8 up, 8 in 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: osdmap e249: 8 total, 8 up, 8 in 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: osdmap e248: 8 total, 8 up, 8 in 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2838003500' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]: dispatch 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi089-79447-48"}]': finished 2024-03-21T13:17:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: osdmap e249: 8 total, 8 up, 8 in 2024-03-21T13:17:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]: dispatch 2024-03-21T13:17:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]: dispatch 2024-03-21T13:17:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: pgmap v338: 228 pgs: 228 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-21T13:17:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]': finished 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: osdmap e250: 8 total, 8 up, 8 in 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: pgmap v338: 228 pgs: 228 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]': finished 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: osdmap e250: 8 total, 8 up, 8 in 2024-03-21T13:17:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: pgmap v338: 228 pgs: 228 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3534936313' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi089-79426-38"}]': finished 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-29"}]': finished 2024-03-21T13:17:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi089-79447-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: osdmap e250: 8 total, 8 up, 8 in 2024-03-21T13:17:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:15.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi089-79426-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi089-79426-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi089-79426-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi089-79426-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi089-79426-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi089-79426-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:16 smithi152 ceph-mon[34051]: pgmap v341: 228 pgs: 228 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T13:17:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:16 smithi152 ceph-mon[34051]: osdmap e251: 8 total, 8 up, 8 in 2024-03-21T13:17:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi089-79426-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:17.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[30095]: pgmap v341: 228 pgs: 228 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T13:17:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[30095]: osdmap e251: 8 total, 8 up, 8 in 2024-03-21T13:17:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi089-79426-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[39209]: pgmap v341: 228 pgs: 228 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T13:17:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[39209]: osdmap e251: 8 total, 8 up, 8 in 2024-03-21T13:17:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi089-79426-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi089-79447-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:17.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:17:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:18.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:17:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: pgmap v344: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: osdmap e252: 8 total, 8 up, 8 in 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi089-79426-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: osdmap e253: 8 total, 8 up, 8 in 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: pgmap v344: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: osdmap e252: 8 total, 8 up, 8 in 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi089-79426-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: osdmap e253: 8 total, 8 up, 8 in 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: pgmap v344: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: osdmap e252: 8 total, 8 up, 8 in 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi089-79426-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: osdmap e253: 8 total, 8 up, 8 in 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:18.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: osdmap e254: 8 total, 8 up, 8 in 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]': finished 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: osdmap e255: 8 total, 8 up, 8 in 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: osdmap e254: 8 total, 8 up, 8 in 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]': finished 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: osdmap e255: 8 total, 8 up, 8 in 2024-03-21T13:17:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: osdmap e254: 8 total, 8 up, 8 in 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-31"}]': finished 2024-03-21T13:17:20.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: osdmap e255: 8 total, 8 up, 8 in 2024-03-21T13:17:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]: dispatch 2024-03-21T13:17:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]: dispatch 2024-03-21T13:17:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1872358284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:20.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]: dispatch 2024-03-21T13:17:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: pgmap v347: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]': finished 2024-03-21T13:17:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: osdmap e256: 8 total, 8 up, 8 in 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: pgmap v347: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]': finished 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: osdmap e256: 8 total, 8 up, 8 in 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: pgmap v347: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-31", "mode": "writeback"}]': finished 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi089-79447-49"}]': finished 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: osdmap e256: 8 total, 8 up, 8 in 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]: dispatch 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:21.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:21.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: osdmap e257: 8 total, 8 up, 8 in 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:22.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi089-79426-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: osdmap e257: 8 total, 8 up, 8 in 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi089-79426-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/404992810' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi089-79426-39"}]': finished 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi089-79447-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: osdmap e257: 8 total, 8 up, 8 in 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:22.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]: dispatch 2024-03-21T13:17:22.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:22.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:22.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:22.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi089-79426-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:22 smithi152 ceph-mon[34051]: pgmap v350: 228 pgs: 18 unknown, 210 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:17:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:22 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]': finished 2024-03-21T13:17:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:22 smithi089 ceph-mon[30095]: pgmap v350: 228 pgs: 18 unknown, 210 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:17:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:22 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]': finished 2024-03-21T13:17:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:22 smithi089 ceph-mon[39209]: pgmap v350: 228 pgs: 18 unknown, 210 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:17:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:22 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-31"}]': finished 2024-03-21T13:17:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi089-79426-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:23 smithi152 ceph-mon[34051]: osdmap e258: 8 total, 8 up, 8 in 2024-03-21T13:17:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi089-79426-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi089-79426-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[30095]: osdmap e258: 8 total, 8 up, 8 in 2024-03-21T13:17:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi089-79426-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi089-79426-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[39209]: osdmap e258: 8 total, 8 up, 8 in 2024-03-21T13:17:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi089-79426-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi089-79447-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:25.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:24 smithi152 ceph-mon[34051]: pgmap v353: 228 pgs: 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:17:25.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:24 smithi152 ceph-mon[34051]: osdmap e259: 8 total, 8 up, 8 in 2024-03-21T13:17:25.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:24 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:25.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi089-79426-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:25.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:24 smithi152 ceph-mon[34051]: osdmap e260: 8 total, 8 up, 8 in 2024-03-21T13:17:25.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:25.223 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[39209]: pgmap v353: 228 pgs: 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[39209]: osdmap e259: 8 total, 8 up, 8 in 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi089-79426-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[39209]: osdmap e260: 8 total, 8 up, 8 in 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:25.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[30095]: pgmap v353: 228 pgs: 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:17:25.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[30095]: osdmap e259: 8 total, 8 up, 8 in 2024-03-21T13:17:25.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:25.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi089-79426-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:25.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[30095]: osdmap e260: 8 total, 8 up, 8 in 2024-03-21T13:17:25.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:25.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:26 smithi152 ceph-mon[34051]: pgmap v356: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:26 smithi152 ceph-mon[34051]: osdmap e261: 8 total, 8 up, 8 in 2024-03-21T13:17:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:27.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:27.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[30095]: pgmap v356: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[30095]: osdmap e261: 8 total, 8 up, 8 in 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[39209]: pgmap v356: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:17:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[39209]: osdmap e261: 8 total, 8 up, 8 in 2024-03-21T13:17:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:17:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: osdmap e262: 8 total, 8 up, 8 in 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:27.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]': finished 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[30095]: osdmap e263: 8 total, 8 up, 8 in 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: osdmap e262: 8 total, 8 up, 8 in 2024-03-21T13:17:27.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]': finished 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:27 smithi089 ceph-mon[39209]: osdmap e263: 8 total, 8 up, 8 in 2024-03-21T13:17:27.927 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: osdmap e262: 8 total, 8 up, 8 in 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2351305009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]: dispatch 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-33"}]': finished 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi089-79447-50"}]': finished 2024-03-21T13:17:27.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-mon[34051]: osdmap e263: 8 total, 8 up, 8 in 2024-03-21T13:17:28.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:17:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:17:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: pgmap v359: 228 pgs: 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]: dispatch 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]: dispatch 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]': finished 2024-03-21T13:17:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:28 smithi152 ceph-mon[34051]: osdmap e264: 8 total, 8 up, 8 in 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: pgmap v359: 228 pgs: 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]: dispatch 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]: dispatch 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]': finished 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[39209]: osdmap e264: 8 total, 8 up, 8 in 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: pgmap v359: 228 pgs: 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]: dispatch 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]: dispatch 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]: dispatch 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:29.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4202056482' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi089-79426-40"}]': finished 2024-03-21T13:17:29.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-33", "mode": "writeback"}]': finished 2024-03-21T13:17:29.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:28 smithi089 ceph-mon[30095]: osdmap e264: 8 total, 8 up, 8 in 2024-03-21T13:17:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2196606036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:17:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: osdmap e265: 8 total, 8 up, 8 in 2024-03-21T13:17:30.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2196606036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: osdmap e265: 8 total, 8 up, 8 in 2024-03-21T13:17:30.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2196606036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:17:30.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:17:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi089-79447-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi089-79426-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: osdmap e265: 8 total, 8 up, 8 in 2024-03-21T13:17:30.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]: dispatch 2024-03-21T13:17:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:30 smithi152 ceph-mon[34051]: pgmap v362: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:30 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:30 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:31.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]': finished 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[30095]: pgmap v362: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]': finished 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[39209]: pgmap v362: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:31.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:31.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:31.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:31.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-33"}]': finished 2024-03-21T13:17:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: osdmap e266: 8 total, 8 up, 8 in 2024-03-21T13:17:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: osdmap e267: 8 total, 8 up, 8 in 2024-03-21T13:17:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi089-79447-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: osdmap e266: 8 total, 8 up, 8 in 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: osdmap e267: 8 total, 8 up, 8 in 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi089-79447-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: osdmap e266: 8 total, 8 up, 8 in 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:32.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi089-79426-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:32.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:32.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: osdmap e267: 8 total, 8 up, 8 in 2024-03-21T13:17:32.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi089-79447-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:32 smithi152 ceph-mon[34051]: pgmap v365: 228 pgs: 228 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:32 smithi152 ceph-mon[34051]: osdmap e268: 8 total, 8 up, 8 in 2024-03-21T13:17:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[30095]: pgmap v365: 228 pgs: 228 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[30095]: osdmap e268: 8 total, 8 up, 8 in 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[39209]: pgmap v365: 228 pgs: 228 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[39209]: osdmap e268: 8 total, 8 up, 8 in 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: pgmap v368: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:17:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi089-79447-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: osdmap e269: 8 total, 8 up, 8 in 2024-03-21T13:17:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: pgmap v368: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi089-79447-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: osdmap e269: 8 total, 8 up, 8 in 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: pgmap v368: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi089-79447-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: osdmap e269: 8 total, 8 up, 8 in 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:36.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:36.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:36.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: osdmap e270: 8 total, 8 up, 8 in 2024-03-21T13:17:36.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: osdmap e271: 8 total, 8 up, 8 in 2024-03-21T13:17:36.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: osdmap e270: 8 total, 8 up, 8 in 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:36.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: osdmap e271: 8 total, 8 up, 8 in 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: osdmap e270: 8 total, 8 up, 8 in 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1910493172' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]: dispatch 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi089-79426-41"}]': finished 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: osdmap e271: 8 total, 8 up, 8 in 2024-03-21T13:17:36.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:37.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: pgmap v371: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-21T13:17:37.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]': finished 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:37.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: osdmap e272: 8 total, 8 up, 8 in 2024-03-21T13:17:37.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:37.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: pgmap v371: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-21T13:17:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]: dispatch 2024-03-21T13:17:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:37.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]': finished 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: osdmap e272: 8 total, 8 up, 8 in 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:37.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: pgmap v371: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:37.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-35", "mode": "writeback"}]': finished 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi089-79426-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: osdmap e272: 8 total, 8 up, 8 in 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]: dispatch 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.078 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:37.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:17:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:17:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:38.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:17:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:17:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: pgmap v374: 228 pgs: 228 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: osdmap e273: 8 total, 8 up, 8 in 2024-03-21T13:17:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:39.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: pgmap v374: 228 pgs: 228 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: osdmap e273: 8 total, 8 up, 8 in 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:39.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: pgmap v374: 228 pgs: 228 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1466759040' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi089-79447-52"}]': finished 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: osdmap e273: 8 total, 8 up, 8 in 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:39.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:40.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:40.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: osdmap e274: 8 total, 8 up, 8 in 2024-03-21T13:17:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:40.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: osdmap e274: 8 total, 8 up, 8 in 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:40.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi089-79426-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: osdmap e274: 8 total, 8 up, 8 in 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:40.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:40.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[30095]: pgmap v377: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[30095]: osdmap e275: 8 total, 8 up, 8 in 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[39209]: pgmap v377: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[39209]: osdmap e275: 8 total, 8 up, 8 in 2024-03-21T13:17:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:40 smithi152 ceph-mon[34051]: pgmap v377: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:40 smithi152 ceph-mon[34051]: osdmap e275: 8 total, 8 up, 8 in 2024-03-21T13:17:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:41 smithi152 ceph-mon[34051]: osdmap e276: 8 total, 8 up, 8 in 2024-03-21T13:17:42.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:42.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:42.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[39209]: osdmap e276: 8 total, 8 up, 8 in 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi089-79447-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[30095]: osdmap e276: 8 total, 8 up, 8 in 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:42.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:42.708 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6785 ms) 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6968 ms) 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7036 ms) 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6981 ms) 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7048 ms) 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7037 ms) 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-21T13:17:42.709 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7022 ms) 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7037 ms) 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6873 ms) 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7031 ms) 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6966 ms) 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7023 ms) 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6968 ms) 2024-03-21T13:17:42.710 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7032 ms) 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7020 ms) 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7043 ms) 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (111870 ms total) 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [----------] Global test environment tear-down 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (210547 ms total) 2024-03-21T13:17:42.711 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-21T13:17:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: pgmap v380: 236 pgs: 3 creating+peering, 5 unknown, 228 active+clean; 4.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:43.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: osdmap e277: 8 total, 8 up, 8 in 2024-03-21T13:17:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:43.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: pgmap v380: 236 pgs: 3 creating+peering, 5 unknown, 228 active+clean; 4.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: osdmap e277: 8 total, 8 up, 8 in 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:43.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: pgmap v380: 236 pgs: 3 creating+peering, 5 unknown, 228 active+clean; 4.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: osdmap e277: 8 total, 8 up, 8 in 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4270430132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]: dispatch 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:43.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:43 smithi152 ceph-mon[34051]: osdmap e278: 8 total, 8 up, 8 in 2024-03-21T13:17:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:44.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:44.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[39209]: osdmap e278: 8 total, 8 up, 8 in 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi089-79426-42"}]': finished 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:44.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[30095]: osdmap e278: 8 total, 8 up, 8 in 2024-03-21T13:17:44.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:44.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:44.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]: dispatch 2024-03-21T13:17:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: pgmap v383: 228 pgs: 228 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:17:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:45.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: osdmap e279: 8 total, 8 up, 8 in 2024-03-21T13:17:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:45.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: pgmap v383: 228 pgs: 228 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: osdmap e279: 8 total, 8 up, 8 in 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: pgmap v383: 228 pgs: 228 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:45.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-35"}]': finished 2024-03-21T13:17:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: osdmap e279: 8 total, 8 up, 8 in 2024-03-21T13:17:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2621382957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:45.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]: dispatch 2024-03-21T13:17:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: osdmap e280: 8 total, 8 up, 8 in 2024-03-21T13:17:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:46.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: osdmap e280: 8 total, 8 up, 8 in 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi089-79447-53"}]': finished 2024-03-21T13:17:46.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: osdmap e280: 8 total, 8 up, 8 in 2024-03-21T13:17:46.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:46.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:46.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:47.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:46 smithi152 ceph-mon[34051]: pgmap v386: 196 pgs: 196 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:17:47.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:47.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:46 smithi152 ceph-mon[34051]: osdmap e281: 8 total, 8 up, 8 in 2024-03-21T13:17:47.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:47.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:47.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:47.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[39209]: pgmap v386: 196 pgs: 196 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[39209]: osdmap e281: 8 total, 8 up, 8 in 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[30095]: pgmap v386: 196 pgs: 196 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi089-79447-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[30095]: osdmap e281: 8 total, 8 up, 8 in 2024-03-21T13:17:47.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:47.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:47.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:47.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:47.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:17:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:48.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:17:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: pgmap v389: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: osdmap e282: 8 total, 8 up, 8 in 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: osdmap e283: 8 total, 8 up, 8 in 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: pgmap v389: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: osdmap e282: 8 total, 8 up, 8 in 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: osdmap e283: 8 total, 8 up, 8 in 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: pgmap v389: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: osdmap e282: 8 total, 8 up, 8 in 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi089-79447-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: osdmap e283: 8 total, 8 up, 8 in 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:48.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]': finished 2024-03-21T13:17:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: osdmap e284: 8 total, 8 up, 8 in 2024-03-21T13:17:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]: dispatch 2024-03-21T13:17:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]: dispatch 2024-03-21T13:17:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]': finished 2024-03-21T13:17:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: osdmap e285: 8 total, 8 up, 8 in 2024-03-21T13:17:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:50.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]': finished 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: osdmap e284: 8 total, 8 up, 8 in 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]: dispatch 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]: dispatch 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]': finished 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: osdmap e285: 8 total, 8 up, 8 in 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:50.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-37"}]': finished 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: osdmap e284: 8 total, 8 up, 8 in 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]: dispatch 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]: dispatch 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-37", "mode": "writeback"}]': finished 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: osdmap e285: 8 total, 8 up, 8 in 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:50.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: pgmap v392: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 4.6 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:51.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: osdmap e286: 8 total, 8 up, 8 in 2024-03-21T13:17:51.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:51.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:51.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:51.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: pgmap v392: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 4.6 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: osdmap e286: 8 total, 8 up, 8 in 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: pgmap v392: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 4.6 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: osdmap e286: 8 total, 8 up, 8 in 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/277995170' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]: dispatch 2024-03-21T13:17:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]: dispatch 2024-03-21T13:17:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]': finished 2024-03-21T13:17:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: osdmap e287: 8 total, 8 up, 8 in 2024-03-21T13:17:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:52.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]': finished 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: osdmap e287: 8 total, 8 up, 8 in 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-37"}]': finished 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi089-79447-54"}]': finished 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: osdmap e287: 8 total, 8 up, 8 in 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:52.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:17:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:52 smithi152 ceph-mon[34051]: pgmap v395: 228 pgs: 21 creating+peering, 207 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:52 smithi152 ceph-mon[34051]: osdmap e288: 8 total, 8 up, 8 in 2024-03-21T13:17:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:53.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[39209]: pgmap v395: 228 pgs: 21 creating+peering, 207 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[39209]: osdmap e288: 8 total, 8 up, 8 in 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[30095]: pgmap v395: 228 pgs: 21 creating+peering, 207 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi089-79447-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[30095]: osdmap e288: 8 total, 8 up, 8 in 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:54 smithi152 ceph-mon[34051]: pgmap v398: 196 pgs: 196 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:17:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:54 smithi152 ceph-mon[34051]: osdmap e289: 8 total, 8 up, 8 in 2024-03-21T13:17:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:55.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[39209]: pgmap v398: 196 pgs: 196 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:17:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[39209]: osdmap e289: 8 total, 8 up, 8 in 2024-03-21T13:17:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[30095]: pgmap v398: 196 pgs: 196 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:17:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[30095]: osdmap e289: 8 total, 8 up, 8 in 2024-03-21T13:17:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:17:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:17:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:55 smithi152 ceph-mon[34051]: osdmap e290: 8 total, 8 up, 8 in 2024-03-21T13:17:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[30095]: osdmap e290: 8 total, 8 up, 8 in 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi089-79447-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[39209]: osdmap e290: 8 total, 8 up, 8 in 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:17:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:56 smithi152 ceph-mon[34051]: pgmap v401: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-21T13:17:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:56 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:56 smithi152 ceph-mon[34051]: osdmap e291: 8 total, 8 up, 8 in 2024-03-21T13:17:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:17:57.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[30095]: pgmap v401: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[30095]: osdmap e291: 8 total, 8 up, 8 in 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[39209]: pgmap v401: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:17:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[39209]: osdmap e291: 8 total, 8 up, 8 in 2024-03-21T13:17:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:17:57.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:17:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:17:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:17:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]': finished 2024-03-21T13:17:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]: dispatch 2024-03-21T13:17:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: osdmap e292: 8 total, 8 up, 8 in 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]': finished 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: osdmap e292: 8 total, 8 up, 8 in 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:57.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:57.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:57 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-39"}]': finished 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]: dispatch 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: osdmap e292: 8 total, 8 up, 8 in 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]: dispatch 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:17:57.958 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:17:58.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:17:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:17:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: pgmap v404: 228 pgs: 228 active+clean; 4.6 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]': finished 2024-03-21T13:17:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:17:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: osdmap e293: 8 total, 8 up, 8 in 2024-03-21T13:17:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:59.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: pgmap v404: 228 pgs: 228 active+clean; 4.6 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]': finished 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: osdmap e293: 8 total, 8 up, 8 in 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: pgmap v404: 228 pgs: 228 active+clean; 4.6 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-39", "mode": "writeback"}]': finished 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: osdmap e293: 8 total, 8 up, 8 in 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2429970577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]: dispatch 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:17:59.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:18:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: osdmap e294: 8 total, 8 up, 8 in 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi089-79447-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]': finished 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi089-79447-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: osdmap e295: 8 total, 8 up, 8 in 2024-03-21T13:18:00.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:17:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi089-79447-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: osdmap e294: 8 total, 8 up, 8 in 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi089-79447-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]': finished 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi089-79447-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: osdmap e295: 8 total, 8 up, 8 in 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi089-79447-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi089-79447-55"}]': finished 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: osdmap e294: 8 total, 8 up, 8 in 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]: dispatch 2024-03-21T13:18:00.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi089-79447-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:00.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:18:00.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-39"}]': finished 2024-03-21T13:18:00.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi089-79447-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:00.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: osdmap e295: 8 total, 8 up, 8 in 2024-03-21T13:18:00.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:17:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi089-79447-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: seed 79633 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (253 ms) 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (18245 ms) 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (24 ms) 2024-03-21T13:18:00.904 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (18522 ms total) 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (9144 ms) 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8961 ms) 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10000 ms) 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: my_snaps [3] 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: my_snaps [4,3] 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-21T13:18:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: promoting some heads 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: waiting for scrubs... 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: done waiting 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46016 ms) 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10028 ms) 2024-03-21T13:18:00.906 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7991 ms) 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8050 ms) 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8276 ms) 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9804 ms) 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8877 ms) 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10061 ms) 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12212 ms) 2024-03-21T13:18:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7977 ms) 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7792 ms) 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13052 ms) 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7911 ms) 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-21T13:18:00.908 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8287 ms) 2024-03-21T13:18:01.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:00 smithi152 ceph-mon[34051]: pgmap v407: 228 pgs: 228 active+clean; 4.6 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:01.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:00 smithi089 ceph-mon[30095]: pgmap v407: 228 pgs: 228 active+clean; 4.6 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:01.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:00 smithi089 ceph-mon[39209]: pgmap v407: 228 pgs: 228 active+clean; 4.6 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:01 smithi152 ceph-mon[34051]: pgmap v410: 196 pgs: 196 active+clean; 4.6 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:01 smithi152 ceph-mon[34051]: osdmap e296: 8 total, 8 up, 8 in 2024-03-21T13:18:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi089-79447-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:01 smithi152 ceph-mon[34051]: osdmap e297: 8 total, 8 up, 8 in 2024-03-21T13:18:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:02.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[30095]: pgmap v410: 196 pgs: 196 active+clean; 4.6 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[30095]: osdmap e296: 8 total, 8 up, 8 in 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi089-79447-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[30095]: osdmap e297: 8 total, 8 up, 8 in 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[39209]: pgmap v410: 196 pgs: 196 active+clean; 4.6 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[39209]: osdmap e296: 8 total, 8 up, 8 in 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi089-79447-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[39209]: osdmap e297: 8 total, 8 up, 8 in 2024-03-21T13:18:02.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:02.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:03 smithi152 ceph-mon[34051]: osdmap e298: 8 total, 8 up, 8 in 2024-03-21T13:18:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:04.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[39209]: osdmap e298: 8 total, 8 up, 8 in 2024-03-21T13:18:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[30095]: osdmap e298: 8 total, 8 up, 8 in 2024-03-21T13:18:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:04.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:04 smithi152 ceph-mon[34051]: pgmap v413: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-21T13:18:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:04 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:04 smithi152 ceph-mon[34051]: osdmap e299: 8 total, 8 up, 8 in 2024-03-21T13:18:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:05.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[30095]: pgmap v413: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-21T13:18:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[30095]: osdmap e299: 8 total, 8 up, 8 in 2024-03-21T13:18:05.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:05.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[39209]: pgmap v413: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-21T13:18:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:05.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[39209]: osdmap e299: 8 total, 8 up, 8 in 2024-03-21T13:18:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:05.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:06.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:18:06.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:06.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: osdmap e300: 8 total, 8 up, 8 in 2024-03-21T13:18:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:06.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:05 smithi152 ceph-mon[34051]: osdmap e301: 8 total, 8 up, 8 in 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: osdmap e300: 8 total, 8 up, 8 in 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:06.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[30095]: osdmap e301: 8 total, 8 up, 8 in 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: osdmap e300: 8 total, 8 up, 8 in 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]: dispatch 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3372579868' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi089-79447-56"}]': finished 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:06.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:05 smithi089 ceph-mon[39209]: osdmap e301: 8 total, 8 up, 8 in 2024-03-21T13:18:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: pgmap v416: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 977 KiB/s wr, 0 op/s 2024-03-21T13:18:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:18:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:18:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:07.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: pgmap v416: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 977 KiB/s wr, 0 op/s 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:07.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: pgmap v416: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 977 KiB/s wr, 0 op/s 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:07.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:18:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:18:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[30095]: osdmap e302: 8 total, 8 up, 8 in 2024-03-21T13:18:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:18:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[39209]: osdmap e302: 8 total, 8 up, 8 in 2024-03-21T13:18:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.969 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:18:07.969 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi089-79447-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:07.969 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:07 smithi152 ceph-mon[34051]: osdmap e302: 8 total, 8 up, 8 in 2024-03-21T13:18:07.969 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:07.969 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:08.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:18:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:18:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:08 smithi152 ceph-mon[34051]: pgmap v419: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:08 smithi152 ceph-mon[34051]: osdmap e303: 8 total, 8 up, 8 in 2024-03-21T13:18:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:09.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]: dispatch 2024-03-21T13:18:09.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]: dispatch 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[30095]: pgmap v419: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[30095]: osdmap e303: 8 total, 8 up, 8 in 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]: dispatch 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]: dispatch 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[39209]: pgmap v419: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[39209]: osdmap e303: 8 total, 8 up, 8 in 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]: dispatch 2024-03-21T13:18:09.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]: dispatch 2024-03-21T13:18:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]': finished 2024-03-21T13:18:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:09 smithi152 ceph-mon[34051]: osdmap e304: 8 total, 8 up, 8 in 2024-03-21T13:18:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:09 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]': finished 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[30095]: osdmap e304: 8 total, 8 up, 8 in 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi089-79447-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-41"}]': finished 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[39209]: osdmap e304: 8 total, 8 up, 8 in 2024-03-21T13:18:10.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:09 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:10.976 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:10 smithi089 ceph-mon[39209]: pgmap v422: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:10.976 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:10 smithi089 ceph-mon[39209]: osdmap e305: 8 total, 8 up, 8 in 2024-03-21T13:18:10.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:10 smithi089 ceph-mon[30095]: pgmap v422: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:10.977 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:10 smithi089 ceph-mon[30095]: osdmap e305: 8 total, 8 up, 8 in 2024-03-21T13:18:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:10 smithi152 ceph-mon[34051]: pgmap v422: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:10 smithi152 ceph-mon[34051]: osdmap e305: 8 total, 8 up, 8 in 2024-03-21T13:18:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: osdmap e306: 8 total, 8 up, 8 in 2024-03-21T13:18:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:12.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:12.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:11 smithi152 ceph-mon[34051]: osdmap e307: 8 total, 8 up, 8 in 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: osdmap e306: 8 total, 8 up, 8 in 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[30095]: osdmap e307: 8 total, 8 up, 8 in 2024-03-21T13:18:12.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: osdmap e306: 8 total, 8 up, 8 in 2024-03-21T13:18:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:12.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:11 smithi089 ceph-mon[39209]: osdmap e307: 8 total, 8 up, 8 in 2024-03-21T13:18:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: pgmap v425: 228 pgs: 9 creating+peering, 23 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi089-79633-6","var": "pg_num","format": "json"}]: dispatch 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: osdmap e308: 8 total, 8 up, 8 in 2024-03-21T13:18:13.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: pgmap v425: 228 pgs: 9 creating+peering, 23 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi089-79633-6","var": "pg_num","format": "json"}]: dispatch 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:13.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: osdmap e308: 8 total, 8 up, 8 in 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: pgmap v425: 228 pgs: 9 creating+peering, 23 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1820940741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi089-79633-6","var": "pg_num","format": "json"}]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi089-79447-57"}]': finished 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:13.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-21T13:18:13.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: osdmap e308: 8 total, 8 up, 8 in 2024-03-21T13:18:13.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-21T13:18:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]': finished 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: osdmap e309: 8 total, 8 up, 8 in 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]': finished 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: osdmap e309: 8 total, 8 up, 8 in 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_count","val": "8"}]': finished 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi089-79447-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: osdmap e309: 8 total, 8 up, 8 in 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:14.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:14.077 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:14 smithi152 ceph-mon[34051]: pgmap v428: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:15.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:14 smithi089 ceph-mon[30095]: pgmap v428: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:14 smithi089 ceph-mon[39209]: pgmap v428: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:15.956 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-21T13:18:15.956 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-21T13:18:15.956 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: ok, hit_set contains 276:602f83fe:::foo:head 2024-03-21T13:18:15.956 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8817 ms) 2024-03-21T13:18:15.956 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-21T13:18:15.956 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg_num = 32 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 0 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 1 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 2 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 3 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 4 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 5 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 6 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 7 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 8 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 9 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 10 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 11 ls 1711027096,0 2024-03-21T13:18:15.957 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 12 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 13 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 14 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 15 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 16 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 17 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 18 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 19 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 20 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 21 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 22 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 23 ls 1711027096,0 2024-03-21T13:18:15.958 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 24 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 25 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 26 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 27 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 28 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 29 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 30 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg 31 ls 1711027096,0 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: pg_num = 32 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6cac518f:::0:head 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:02547ec2:::1:head 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f905c69b:::2:head 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:cfc208b3:::3:head 2024-03-21T13:18:15.959 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d83876eb:::4:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b29083e3:::5:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c4fdafeb:::6:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:5c6b0b28:::7:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:bd63b0f1:::8:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:e960b815:::9:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:52ea6a34:::10:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:89d3ae78:::11:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:de5d7c5f:::12:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:566253c9:::13:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:62a1935d:::14:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:863748b0:::15:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:3958e169:::16:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:4d4dabf9:::17:head 2024-03-21T13:18:15.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:8391935d:::18:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:28883081:::19:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:69259c59:::20:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:4bdb80b7:::21:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a11c5d71:::22:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:271af37b:::23:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:95b121be:::24:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:58d1031b:::25:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:0a050783:::26:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c709704c:::27:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:cbe56eaf:::28:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:86b4b162:::29:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:70d89383:::30:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:dd450c7c:::31:head 2024-03-21T13:18:15.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6d5729b1:::32:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c388f3fb:::33:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:56cfea31:::34:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:9dbc1bf7:::35:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:40b74ccd:::36:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:4d5aaf42:::37:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:920f362c:::38:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6cc53222:::39:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:9cad833f:::40:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:1ea84d41:::41:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c4480ef6:::42:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a694361e:::43:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d1bd33e9:::44:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:ddc2cd5d:::45:head 2024-03-21T13:18:15.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:2b782207:::46:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:7b187fca:::47:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:90ecdf6f:::48:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a5ed95fe:::49:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:ea0eaa55:::50:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f33ef17b:::51:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a0d1b2f6:::52:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:60c5229e:::53:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:edcbc575:::54:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:102cf253:::55:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:efb7fb0b:::56:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:50d0a326:::57:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d4dc5daf:::58:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:3a130462:::59:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:ec87ed71:::60:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d5bc9454:::61:head 2024-03-21T13:18:15.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:3ddfe313:::62:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:7c2816b9:::63:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:47e00e4d:::64:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c6410c18:::65:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b48ed237:::66:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:cd63ad31:::67:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b179e92b:::68:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:0d9f741a:::69:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6d3352ae:::70:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c6d5c19e:::71:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:bc4729c3:::72:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:77e930b9:::73:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:0abeecfd:::74:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b7c37e15:::75:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b6378398:::76:head 2024-03-21T13:18:15.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:02bd68de:::77:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:cc795d2d:::78:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:630d4fea:::79:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:e0d29ef5:::80:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:fd6f13d2:::81:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:606461d5:::82:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:eadbdc43:::83:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:8761d0bb:::84:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:9ef0186f:::85:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:e0d41294:::86:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:961de695:::87:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:1423148f:::88:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:633a8fa2:::89:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a8653809:::90:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:3dac8b33:::91:head 2024-03-21T13:18:15.965 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:35aad435:::92:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f6dcc343:::93:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:dbbdad87:::94:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:1cb48ce0:::95:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:03cd461c:::96:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:17a4ea99:::97:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:9993c9a7:::98:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6394211c:::99:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:94c7ae57:::100:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6fdee5bb:::101:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:9a477fd1:::102:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:eb850916:::103:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:affc56b9:::104:head 2024-03-21T13:18:15.966 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b42dc814:::105:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f319f8f0:::106:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:9a40b9de:::107:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:8b524f28:::108:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:e3de589f:::109:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:90f90a5b:::110:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a7b4f1d7:::111:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:af51766e:::112:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b6f90bd1:::113:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:e0261208:::114:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c9569ef7:::115:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:61bebe50:::116:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:fe93412b:::117:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d3d38bee:::118:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:3100ba0c:::119:head 2024-03-21T13:18:15.967 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d0560ada:::120:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f0ea8b35:::121:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:766f231a:::122:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a07a2582:::123:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:bd7c6b3a:::124:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:fb2ddaff:::125:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:4408e1fe:::126:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:ee1df7a7:::127:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c3002909:::128:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:4f48ffa9:::129:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:edf38733:::130:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c08425c0:::131:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:5f902d98:::132:head 2024-03-21T13:18:15.968 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:41ea2c93:::133:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:813cee13:::134:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:0131818d:::135:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:26ba5a85:::136:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:381b8a5a:::137:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:28797e47:::138:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:bfca7f22:::139:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:36807075:::140:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:80b03975:::141:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:5c15709b:::142:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f39ea15e:::143:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:ea992956:::144:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:48887b1c:::145:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:9f24a9dd:::146:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:987f100b:::147:head 2024-03-21T13:18:15.969 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d2dd3581:::148:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:7fed1808:::149:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c80b70e9:::150:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:85ed90f9:::151:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:36428b24:::152:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d044c34a:::153:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:7c18bf58:::154:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d1c21232:::155:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a7a3c575:::156:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:87da0633:::157:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d5ac3822:::158:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:3f20522d:::159:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6ca26563:::160:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:532ce135:::161:head 2024-03-21T13:18:15.970 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:c78863e6:::162:head 2024-03-21T13:18:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-21T13:18:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: pgmap v431: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: osdmap e310: 8 total, 8 up, 8 in 2024-03-21T13:18:16.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-21T13:18:16.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:16.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-21T13:18:16.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: osdmap e311: 8 total, 8 up, 8 in 2024-03-21T13:18:16.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi089-79633-43","var": "pg_num","format": "json"}]: dispatch 2024-03-21T13:18:16.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: pgmap v431: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: osdmap e310: 8 total, 8 up, 8 in 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: osdmap e311: 8 total, 8 up, 8 in 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi089-79633-43","var": "pg_num","format": "json"}]: dispatch 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: pgmap v431: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: osdmap e310: 8 total, 8 up, 8 in 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-21T13:18:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi089-79447-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-21T13:18:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: osdmap e311: 8 total, 8 up, 8 in 2024-03-21T13:18:16.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi089-79633-43","var": "pg_num","format": "json"}]: dispatch 2024-03-21T13:18:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]: dispatch 2024-03-21T13:18:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]: dispatch 2024-03-21T13:18:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]': finished 2024-03-21T13:18:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:16 smithi152 ceph-mon[34051]: osdmap e312: 8 total, 8 up, 8 in 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]': finished 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[30095]: osdmap e312: 8 total, 8 up, 8 in 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]: dispatch 2024-03-21T13:18:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-43"}]': finished 2024-03-21T13:18:17.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:16 smithi089 ceph-mon[39209]: osdmap e312: 8 total, 8 up, 8 in 2024-03-21T13:18:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:18:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:18.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:18:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:18:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:17 smithi152 ceph-mon[34051]: pgmap v434: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 KiB/s wr, 29 op/s 2024-03-21T13:18:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:17 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:17 smithi152 ceph-mon[34051]: osdmap e313: 8 total, 8 up, 8 in 2024-03-21T13:18:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:18.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[30095]: pgmap v434: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 KiB/s wr, 29 op/s 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[30095]: osdmap e313: 8 total, 8 up, 8 in 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[39209]: pgmap v434: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 KiB/s wr, 29 op/s 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[39209]: osdmap e313: 8 total, 8 up, 8 in 2024-03-21T13:18:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:18.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:19 smithi152 ceph-mon[34051]: osdmap e314: 8 total, 8 up, 8 in 2024-03-21T13:18:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:20.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[30095]: osdmap e314: 8 total, 8 up, 8 in 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[39209]: osdmap e314: 8 total, 8 up, 8 in 2024-03-21T13:18:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4292983619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]: dispatch 2024-03-21T13:18:20.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: pgmap v437: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: osdmap e315: 8 total, 8 up, 8 in 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:21.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: pgmap v437: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: osdmap e315: 8 total, 8 up, 8 in 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: pgmap v437: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi089-79447-58"}]': finished 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: osdmap e315: 8 total, 8 up, 8 in 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:21.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:18:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: osdmap e316: 8 total, 8 up, 8 in 2024-03-21T13:18:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:18:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]': finished 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: osdmap e316: 8 total, 8 up, 8 in 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]': finished 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:18:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: osdmap e316: 8 total, 8 up, 8 in 2024-03-21T13:18:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:18:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_count","val": "3"}]': finished 2024-03-21T13:18:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: pgmap v440: 228 pgs: 21 unknown, 207 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:18:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: osdmap e317: 8 total, 8 up, 8 in 2024-03-21T13:18:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:18:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]': finished 2024-03-21T13:18:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: osdmap e318: 8 total, 8 up, 8 in 2024-03-21T13:18:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:23.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: pgmap v440: 228 pgs: 21 unknown, 207 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: osdmap e317: 8 total, 8 up, 8 in 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]': finished 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: osdmap e318: 8 total, 8 up, 8 in 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: pgmap v440: 228 pgs: 21 unknown, 207 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:18:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: osdmap e317: 8 total, 8 up, 8 in 2024-03-21T13:18:23.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:18:23.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi089-79447-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:23.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_period","val": "3"}]': finished 2024-03-21T13:18:23.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: osdmap e318: 8 total, 8 up, 8 in 2024-03-21T13:18:23.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:23.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:24 smithi152 ceph-mon[34051]: pgmap v443: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:24 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:18:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:24 smithi152 ceph-mon[34051]: osdmap e319: 8 total, 8 up, 8 in 2024-03-21T13:18:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:18:24.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[30095]: pgmap v443: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[30095]: osdmap e319: 8 total, 8 up, 8 in 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[39209]: pgmap v443: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:18:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[39209]: osdmap e319: 8 total, 8 up, 8 in 2024-03-21T13:18:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:18:24.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:18:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-21T13:18:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:25 smithi152 ceph-mon[34051]: osdmap e320: 8 total, 8 up, 8 in 2024-03-21T13:18:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:26.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-21T13:18:26.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[39209]: osdmap e320: 8 total, 8 up, 8 in 2024-03-21T13:18:26.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:26.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-21T13:18:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[30095]: osdmap e320: 8 total, 8 up, 8 in 2024-03-21T13:18:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:26.771 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:7462ddf6:::.RoundTripAppendPP (3009 ms) 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3233 ms) 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2785 ms) 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2998 ms) 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: setting pool EIO 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: max_success 100, min_failed 101 2024-03-21T13:18:26.772 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3979 ms) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3021 ms) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (127453 ms total) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3008 ms) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9062 ms) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15062 ms) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2853 ms) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29986 ms total) 2024-03-21T13:18:26.773 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3009 ms) 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3009 ms total) 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14001 ms) 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7028 ms) 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6967 ms) 2024-03-21T13:18:26.774 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7028 ms) 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3254 ms) 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7019 ms) 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6812 ms) 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6895 ms) 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7044 ms) 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7026 ms) 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-21T13:18:26.775 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7045 ms) 2024-03-21T13:18:26.776 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-21T13:18:26.776 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7025 ms) 2024-03-21T13:18:26.776 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-21T13:18:26.776 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7016 ms) 2024-03-21T13:18:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:26 smithi152 ceph-mon[34051]: pgmap v446: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:27.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:26 smithi152 ceph-mon[34051]: osdmap e321: 8 total, 8 up, 8 in 2024-03-21T13:18:27.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:27.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:27.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[30095]: pgmap v446: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[30095]: osdmap e321: 8 total, 8 up, 8 in 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[39209]: pgmap v446: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[39209]: osdmap e321: 8 total, 8 up, 8 in 2024-03-21T13:18:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3976437130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]: dispatch 2024-03-21T13:18:27.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi089-79447-59"}]': finished 2024-03-21T13:18:27.778 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:18:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:18:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:27 smithi152 ceph-mon[34051]: osdmap e322: 8 total, 8 up, 8 in 2024-03-21T13:18:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:28.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:18:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:18:28.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:27 smithi089 ceph-mon[30095]: osdmap e322: 8 total, 8 up, 8 in 2024-03-21T13:18:28.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:27 smithi089 ceph-mon[39209]: osdmap e322: 8 total, 8 up, 8 in 2024-03-21T13:18:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:28 smithi152 ceph-mon[34051]: pgmap v449: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:18:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:28 smithi152 ceph-mon[34051]: osdmap e323: 8 total, 8 up, 8 in 2024-03-21T13:18:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2197912958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:28 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[30095]: pgmap v449: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:18:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[30095]: osdmap e323: 8 total, 8 up, 8 in 2024-03-21T13:18:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2197912958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[39209]: pgmap v449: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:18:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[39209]: osdmap e323: 8 total, 8 up, 8 in 2024-03-21T13:18:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2197912958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:29.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:28 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:29 smithi152 ceph-mon[34051]: osdmap e324: 8 total, 8 up, 8 in 2024-03-21T13:18:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:29 smithi152 ceph-mon[34051]: pgmap v452: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:18:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:29 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:18:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:29 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:30.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[39209]: osdmap e324: 8 total, 8 up, 8 in 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[39209]: pgmap v452: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi089-79447-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[30095]: osdmap e324: 8 total, 8 up, 8 in 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[30095]: pgmap v452: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:18:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:29 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:31.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: osdmap e325: 8 total, 8 up, 8 in 2024-03-21T13:18:31.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:18:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:18:31.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:30 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: osdmap e325: 8 total, 8 up, 8 in 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:18:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: osdmap e325: 8 total, 8 up, 8 in 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:18:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:30 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:18:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:31 smithi152 ceph-mon[34051]: osdmap e326: 8 total, 8 up, 8 in 2024-03-21T13:18:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:32.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:31 smithi152 ceph-mon[34051]: pgmap v455: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-21T13:18:32.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[30095]: osdmap e326: 8 total, 8 up, 8 in 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[30095]: pgmap v455: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi089-79447-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[39209]: osdmap e326: 8 total, 8 up, 8 in 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:32.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:31 smithi089 ceph-mon[39209]: pgmap v455: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-21T13:18:33.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:32 smithi152 ceph-mon[34051]: osdmap e327: 8 total, 8 up, 8 in 2024-03-21T13:18:33.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:32 smithi089 ceph-mon[30095]: osdmap e327: 8 total, 8 up, 8 in 2024-03-21T13:18:33.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:32 smithi089 ceph-mon[39209]: osdmap e327: 8 total, 8 up, 8 in 2024-03-21T13:18:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:33 smithi089 ceph-mon[30095]: osdmap e328: 8 total, 8 up, 8 in 2024-03-21T13:18:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:33 smithi089 ceph-mon[30095]: pgmap v458: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-21T13:18:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:33 smithi089 ceph-mon[39209]: osdmap e328: 8 total, 8 up, 8 in 2024-03-21T13:18:34.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:33 smithi089 ceph-mon[39209]: pgmap v458: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-21T13:18:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi089-79447-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:33 smithi152 ceph-mon[34051]: osdmap e328: 8 total, 8 up, 8 in 2024-03-21T13:18:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:33 smithi152 ceph-mon[34051]: pgmap v458: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-21T13:18:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:34 smithi089 ceph-mon[30095]: osdmap e329: 8 total, 8 up, 8 in 2024-03-21T13:18:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:34 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:35.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:34 smithi089 ceph-mon[39209]: osdmap e329: 8 total, 8 up, 8 in 2024-03-21T13:18:35.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:34 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:34 smithi152 ceph-mon[34051]: osdmap e329: 8 total, 8 up, 8 in 2024-03-21T13:18:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:34 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:35 smithi152 ceph-mon[34051]: osdmap e330: 8 total, 8 up, 8 in 2024-03-21T13:18:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:35 smithi152 ceph-mon[34051]: pgmap v461: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:36.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[30095]: osdmap e330: 8 total, 8 up, 8 in 2024-03-21T13:18:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:36.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[30095]: pgmap v461: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[39209]: osdmap e330: 8 total, 8 up, 8 in 2024-03-21T13:18:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:35 smithi089 ceph-mon[39209]: pgmap v461: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: osdmap e331: 8 total, 8 up, 8 in 2024-03-21T13:18:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:37.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:37.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]: dispatch 2024-03-21T13:18:37.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]: dispatch 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: osdmap e331: 8 total, 8 up, 8 in 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]: dispatch 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]: dispatch 2024-03-21T13:18:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:37.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: osdmap e331: 8 total, 8 up, 8 in 2024-03-21T13:18:37.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/922931277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:37.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]: dispatch 2024-03-21T13:18:37.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:37.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:37.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]: dispatch 2024-03-21T13:18:37.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]: dispatch 2024-03-21T13:18:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:18:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:38.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:18:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]': finished 2024-03-21T13:18:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-mon[34051]: osdmap e332: 8 total, 8 up, 8 in 2024-03-21T13:18:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-mon[34051]: pgmap v464: 228 pgs: 228 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-21T13:18:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:38.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:38.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]': finished 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[30095]: osdmap e332: 8 total, 8 up, 8 in 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[30095]: pgmap v464: 228 pgs: 228 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi089-79447-61"}]': finished 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-45"}]': finished 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[39209]: osdmap e332: 8 total, 8 up, 8 in 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[39209]: pgmap v464: 228 pgs: 228 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-21T13:18:38.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:38.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:38.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:38 smithi152 ceph-mon[34051]: osdmap e333: 8 total, 8 up, 8 in 2024-03-21T13:18:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi089-79447-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:38 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[30095]: osdmap e333: 8 total, 8 up, 8 in 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi089-79447-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[39209]: osdmap e333: 8 total, 8 up, 8 in 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi089-79447-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:38 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:40.226 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[39209]: pgmap v466: 196 pgs: 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[39209]: osdmap e334: 8 total, 8 up, 8 in 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[30095]: pgmap v466: 196 pgs: 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[30095]: osdmap e334: 8 total, 8 up, 8 in 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:40.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:39 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:39 smithi152 ceph-mon[34051]: pgmap v466: 196 pgs: 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:39 smithi152 ceph-mon[34051]: osdmap e334: 8 total, 8 up, 8 in 2024-03-21T13:18:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:39 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi089-79447-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: osdmap e335: 8 total, 8 up, 8 in 2024-03-21T13:18:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:41.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:41.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: osdmap e336: 8 total, 8 up, 8 in 2024-03-21T13:18:41.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:41.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi089-79447-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: osdmap e335: 8 total, 8 up, 8 in 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: osdmap e336: 8 total, 8 up, 8 in 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi089-79447-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: osdmap e335: 8 total, 8 up, 8 in 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: osdmap e336: 8 total, 8 up, 8 in 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:41 smithi152 ceph-mon[34051]: pgmap v469: 236 pgs: 5 creating+peering, 23 unknown, 208 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]': finished 2024-03-21T13:18:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:41 smithi152 ceph-mon[34051]: osdmap e337: 8 total, 8 up, 8 in 2024-03-21T13:18:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]: dispatch 2024-03-21T13:18:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]: dispatch 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[30095]: pgmap v469: 236 pgs: 5 creating+peering, 23 unknown, 208 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]': finished 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[30095]: osdmap e337: 8 total, 8 up, 8 in 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]: dispatch 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]: dispatch 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[39209]: pgmap v469: 236 pgs: 5 creating+peering, 23 unknown, 208 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-47"}]': finished 2024-03-21T13:18:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[39209]: osdmap e337: 8 total, 8 up, 8 in 2024-03-21T13:18:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]: dispatch 2024-03-21T13:18:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:42.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]: dispatch 2024-03-21T13:18:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:18:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]': finished 2024-03-21T13:18:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: osdmap e338: 8 total, 8 up, 8 in 2024-03-21T13:18:43.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:43.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]': finished 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: osdmap e338: 8 total, 8 up, 8 in 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-47", "mode": "writeback"}]': finished 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: osdmap e338: 8 total, 8 up, 8 in 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]: dispatch 2024-03-21T13:18:43.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:18:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:43 smithi152 ceph-mon[34051]: pgmap v472: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:18:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:18:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:43 smithi152 ceph-mon[34051]: osdmap e339: 8 total, 8 up, 8 in 2024-03-21T13:18:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[30095]: pgmap v472: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[30095]: osdmap e339: 8 total, 8 up, 8 in 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[39209]: pgmap v472: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/795978425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi089-79447-62"}]': finished 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[39209]: osdmap e339: 8 total, 8 up, 8 in 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:18:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: osdmap e340: 8 total, 8 up, 8 in 2024-03-21T13:18:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:45.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: osdmap e340: 8 total, 8 up, 8 in 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:45.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:18:45.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi089-79447-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:45.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: osdmap e340: 8 total, 8 up, 8 in 2024-03-21T13:18:45.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:45.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:45.327 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:18:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:46 smithi152 ceph-mon[34051]: pgmap v475: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s wr, 3 op/s 2024-03-21T13:18:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:46 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:18:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:18:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:46 smithi089 ceph-mon[30095]: pgmap v475: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s wr, 3 op/s 2024-03-21T13:18:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:46 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:18:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:18:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:46 smithi089 ceph-mon[39209]: pgmap v475: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s wr, 3 op/s 2024-03-21T13:18:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:46 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:18:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:18:47.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: osdmap e341: 8 total, 8 up, 8 in 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: osdmap e342: 8 total, 8 up, 8 in 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: osdmap e341: 8 total, 8 up, 8 in 2024-03-21T13:18:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:18:47.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:47.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:18:47.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: osdmap e342: 8 total, 8 up, 8 in 2024-03-21T13:18:47.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:18:47.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:18:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:18:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: osdmap e341: 8 total, 8 up, 8 in 2024-03-21T13:18:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:18:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi089-79447-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:18:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: osdmap e342: 8 total, 8 up, 8 in 2024-03-21T13:18:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:18:47.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:18:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:18:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:18:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:48.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:18:47 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:18:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:48 smithi152 ceph-mon[34051]: pgmap v478: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-21T13:18:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:48 smithi152 ceph-mon[34051]: osdmap e343: 8 total, 8 up, 8 in 2024-03-21T13:18:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:18:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[30095]: pgmap v478: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[30095]: osdmap e343: 8 total, 8 up, 8 in 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[39209]: pgmap v478: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-21T13:18:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[39209]: osdmap e343: 8 total, 8 up, 8 in 2024-03-21T13:18:48.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:18:48.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:18:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-21T13:18:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:49 smithi152 ceph-mon[34051]: osdmap e344: 8 total, 8 up, 8 in 2024-03-21T13:18:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:49.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[30095]: osdmap e344: 8 total, 8 up, 8 in 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-21T13:18:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[39209]: osdmap e344: 8 total, 8 up, 8 in 2024-03-21T13:18:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: pgmap v481: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: osdmap e345: 8 total, 8 up, 8 in 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:50.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: pgmap v481: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: osdmap e345: 8 total, 8 up, 8 in 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: pgmap v481: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: osdmap e345: 8 total, 8 up, 8 in 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2861024360' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:50.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]: dispatch 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: pgmap v484: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]': finished 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: osdmap e346: 8 total, 8 up, 8 in 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi089-79447-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:52.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:52.323 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: pgmap v484: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]': finished 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: osdmap e346: 8 total, 8 up, 8 in 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi089-79447-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: pgmap v484: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]': finished 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi089-79447-63"}]': finished 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: osdmap e346: 8 total, 8 up, 8 in 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi089-79447-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-47"}]: dispatch 2024-03-21T13:18:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi089-79447-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:53 smithi152 ceph-mon[34051]: osdmap e347: 8 total, 8 up, 8 in 2024-03-21T13:18:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi089-79447-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:53.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi089-79447-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:53.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:53 smithi089 ceph-mon[30095]: osdmap e347: 8 total, 8 up, 8 in 2024-03-21T13:18:53.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi089-79447-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi089-79447-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:53 smithi089 ceph-mon[39209]: osdmap e347: 8 total, 8 up, 8 in 2024-03-21T13:18:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi089-79447-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: pgmap v487: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:18:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: osdmap e348: 8 total, 8 up, 8 in 2024-03-21T13:18:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi089-79447-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:54.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: osdmap e349: 8 total, 8 up, 8 in 2024-03-21T13:18:54.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:54.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:54.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: pgmap v487: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: osdmap e348: 8 total, 8 up, 8 in 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi089-79447-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: osdmap e349: 8 total, 8 up, 8 in 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: pgmap v487: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: osdmap e348: 8 total, 8 up, 8 in 2024-03-21T13:18:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:54.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:18:54.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi089-79447-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:54.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:18:54.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: osdmap e349: 8 total, 8 up, 8 in 2024-03-21T13:18:54.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:54.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:18:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:55 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:55 smithi152 ceph-mon[34051]: osdmap e350: 8 total, 8 up, 8 in 2024-03-21T13:18:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:18:55.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:18:55.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[30095]: osdmap e350: 8 total, 8 up, 8 in 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[39209]: osdmap e350: 8 total, 8 up, 8 in 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:18:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:18:56.472 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:56 smithi152 ceph-mon[34051]: pgmap v490: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]': finished 2024-03-21T13:18:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:56 smithi152 ceph-mon[34051]: osdmap e351: 8 total, 8 up, 8 in 2024-03-21T13:18:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]: dispatch 2024-03-21T13:18:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]: dispatch 2024-03-21T13:18:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[30095]: pgmap v490: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]': finished 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[30095]: osdmap e351: 8 total, 8 up, 8 in 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]: dispatch 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]: dispatch 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[39209]: pgmap v490: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-49"}]': finished 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[39209]: osdmap e351: 8 total, 8 up, 8 in 2024-03-21T13:18:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]: dispatch 2024-03-21T13:18:56.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]: dispatch 2024-03-21T13:18:56.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:57.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]': finished 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[30095]: osdmap e352: 8 total, 8 up, 8 in 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]': finished 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[39209]: osdmap e352: 8 total, 8 up, 8 in 2024-03-21T13:18:57.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:57 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:18:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-49", "mode": "readproxy"}]': finished 2024-03-21T13:18:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:57 smithi152 ceph-mon[34051]: osdmap e352: 8 total, 8 up, 8 in 2024-03-21T13:18:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]: dispatch 2024-03-21T13:18:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:18:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:18:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:18:58.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:18:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: pgmap v493: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:18:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:18:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: osdmap e353: 8 total, 8 up, 8 in 2024-03-21T13:18:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi089-79447-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: pgmap v493: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: osdmap e353: 8 total, 8 up, 8 in 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi089-79447-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: pgmap v493: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:18:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:18:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:18:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1590970950' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi089-79447-64"}]': finished 2024-03-21T13:18:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: osdmap e353: 8 total, 8 up, 8 in 2024-03-21T13:18:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:58.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi089-79447-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:18:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi089-79447-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:59 smithi152 ceph-mon[34051]: osdmap e354: 8 total, 8 up, 8 in 2024-03-21T13:18:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:18:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi089-79447-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi089-79447-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[30095]: osdmap e354: 8 total, 8 up, 8 in 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi089-79447-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi089-79447-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[39209]: osdmap e354: 8 total, 8 up, 8 in 2024-03-21T13:18:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:18:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi089-79447-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:19:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:00 smithi152 ceph-mon[34051]: pgmap v496: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:19:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:00 smithi152 ceph-mon[34051]: osdmap e355: 8 total, 8 up, 8 in 2024-03-21T13:19:00.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:00 smithi089 ceph-mon[30095]: pgmap v496: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:19:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:00 smithi089 ceph-mon[30095]: osdmap e355: 8 total, 8 up, 8 in 2024-03-21T13:19:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:00 smithi089 ceph-mon[39209]: pgmap v496: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:19:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:00 smithi089 ceph-mon[39209]: osdmap e355: 8 total, 8 up, 8 in 2024-03-21T13:19:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:01 smithi152 ceph-mon[34051]: pgmap v499: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:19:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi089-79447-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:01 smithi152 ceph-mon[34051]: osdmap e356: 8 total, 8 up, 8 in 2024-03-21T13:19:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:01 smithi089 ceph-mon[30095]: pgmap v499: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:19:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi089-79447-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:01 smithi089 ceph-mon[30095]: osdmap e356: 8 total, 8 up, 8 in 2024-03-21T13:19:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:01 smithi089 ceph-mon[39209]: pgmap v499: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:19:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi089-79447-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:01 smithi089 ceph-mon[39209]: osdmap e356: 8 total, 8 up, 8 in 2024-03-21T13:19:03.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:02 smithi152 ceph-mon[34051]: osdmap e357: 8 total, 8 up, 8 in 2024-03-21T13:19:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:02 smithi089 ceph-mon[30095]: osdmap e357: 8 total, 8 up, 8 in 2024-03-21T13:19:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:02 smithi089 ceph-mon[39209]: osdmap e357: 8 total, 8 up, 8 in 2024-03-21T13:19:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:03 smithi152 ceph-mon[34051]: pgmap v502: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:19:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:03 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:03 smithi152 ceph-mon[34051]: osdmap e358: 8 total, 8 up, 8 in 2024-03-21T13:19:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:19:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[30095]: pgmap v502: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:19:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[30095]: osdmap e358: 8 total, 8 up, 8 in 2024-03-21T13:19:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:19:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[39209]: pgmap v502: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:19:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[39209]: osdmap e358: 8 total, 8 up, 8 in 2024-03-21T13:19:04.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:19:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:04 smithi152 ceph-mon[34051]: osdmap e359: 8 total, 8 up, 8 in 2024-03-21T13:19:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:19:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:04 smithi152 ceph-mon[34051]: osdmap e360: 8 total, 8 up, 8 in 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[30095]: osdmap e359: 8 total, 8 up, 8 in 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[30095]: osdmap e360: 8 total, 8 up, 8 in 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[39209]: osdmap e359: 8 total, 8 up, 8 in 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]: dispatch 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2738208863' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi089-79447-65"}]': finished 2024-03-21T13:19:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:04 smithi089 ceph-mon[39209]: osdmap e360: 8 total, 8 up, 8 in 2024-03-21T13:19:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:05 smithi152 ceph-mon[34051]: pgmap v505: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi089-79447-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:19:06.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[30095]: pgmap v505: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi089-79447-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:19:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[39209]: pgmap v505: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi089-79447-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:19:07.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi089-79447-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:19:07.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:07 smithi089 ceph-mon[39209]: osdmap e361: 8 total, 8 up, 8 in 2024-03-21T13:19:07.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi089-79447-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:07.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi089-79447-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:19:07.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:07 smithi089 ceph-mon[30095]: osdmap e361: 8 total, 8 up, 8 in 2024-03-21T13:19:07.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi089-79447-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi089-79447-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:07 smithi152 ceph-mon[34051]: osdmap e361: 8 total, 8 up, 8 in 2024-03-21T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi089-79447-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:19:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:19:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:08.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:19:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: pgmap v508: 228 pgs: 228 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: osdmap e362: 8 total, 8 up, 8 in 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi089-79447-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: osdmap e363: 8 total, 8 up, 8 in 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:08.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: pgmap v508: 228 pgs: 228 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: osdmap e362: 8 total, 8 up, 8 in 2024-03-21T13:19:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi089-79447-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: osdmap e363: 8 total, 8 up, 8 in 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: pgmap v508: 228 pgs: 228 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: osdmap e362: 8 total, 8 up, 8 in 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi089-79447-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:19:08.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: osdmap e363: 8 total, 8 up, 8 in 2024-03-21T13:19:08.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:08.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]': finished 2024-03-21T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: osdmap e364: 8 total, 8 up, 8 in 2024-03-21T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:09.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]': finished 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: osdmap e364: 8 total, 8 up, 8 in 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:09.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:19:09.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]': finished 2024-03-21T13:19:09.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: osdmap e364: 8 total, 8 up, 8 in 2024-03-21T13:19:09.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:09.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:09.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:09.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-49"}]: dispatch 2024-03-21T13:19:09.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2163:head 2024-03-21T13:19:09.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:5d165639:::164:head 2024-03-21T13:19:09.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f43765fc:::165:head 2024-03-21T13:19:09.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b4c720e9:::166:head 2024-03-21T13:19:09.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:e694b040:::167:head 2024-03-21T13:19:09.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:afa38db2:::168:head 2024-03-21T13:19:09.960 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:77ba9f53:::169:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:87495034:::170:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:7c96bf0e:::171:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:dbe346cc:::172:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:e943ec24:::173:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f97a9c0c:::174:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:6f26e74d:::175:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:4f95e106:::176:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:0e6f2f8f:::177:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:05db05f1:::178:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:38a78d66:::179:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:d095610b:::180:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:a1a9d709:::181:head 2024-03-21T13:19:09.961 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:1e5d39db:::182:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:f7df4fb9:::183:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:03a7f161:::184:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:ba70721e:::185:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:28e5662d:::186:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:973d52de:::187:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:4303eb1c:::188:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:b990b48e:::189:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:29b8165b:::190:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:3547f197:::191:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:7e260936:::192:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:1abec7b1:::193:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:10fdda93:::194:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:15817eea:::195:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:770bab57:::196:head 2024-03-21T13:19:09.962 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:ed9e13e7:::197:head 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:71471a8f:::198:head 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: checking for 278:10fb1d02:::199:head 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8024 ms) 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: first is 1711027105 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,1711027110,1711027111,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,1711027110,1711027111,0 2024-03-21T13:19:09.963 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,1711027110,1711027111,0 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,1711027110,1711027111,1711027113,1711027114,0 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,1711027110,1711027111,1711027113,1711027114,0 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027105,1711027107,1711027108,1711027110,1711027111,1711027113,1711027114,0 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027108,1711027110,1711027111,1711027113,1711027114,1711027116,1711027117,0 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: first now 1711027108, trimmed 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20120 ms) 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: foo0 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14035 ms) 2024-03-21T13:19:09.964 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-21T13:19:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[39209]: pgmap v511: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[39209]: osdmap e365: 8 total, 8 up, 8 in 2024-03-21T13:19:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:10.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[30095]: pgmap v511: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:10.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:10.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[30095]: osdmap e365: 8 total, 8 up, 8 in 2024-03-21T13:19:10.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:10 smithi152 ceph-mon[34051]: pgmap v511: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:10 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:10 smithi152 ceph-mon[34051]: osdmap e365: 8 total, 8 up, 8 in 2024-03-21T13:19:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:11.972 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ 2024-03-21T13:19:11.972 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3030 ms) 2024-03-21T13:19:11.972 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7053 ms) 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7031 ms) 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7014 ms) 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7018 ms) 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7020 ms) 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-21T13:19:11.973 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7029 ms) 2024-03-21T13:19:11.974 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139355 ms total) 2024-03-21T13:19:11.974 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: 2024-03-21T13:19:11.974 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-21T13:19:11.974 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (299803 ms total) 2024-03-21T13:19:11.974 INFO:tasks.workunit.client.0.smithi089.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-21T13:19:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:11 smithi152 ceph-mon[34051]: pgmap v514: 196 pgs: 196 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:11 smithi152 ceph-mon[34051]: osdmap e366: 8 total, 8 up, 8 in 2024-03-21T13:19:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:12.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[30095]: pgmap v514: 196 pgs: 196 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[30095]: osdmap e366: 8 total, 8 up, 8 in 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[39209]: pgmap v514: 196 pgs: 196 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[39209]: osdmap e366: 8 total, 8 up, 8 in 2024-03-21T13:19:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]: dispatch 2024-03-21T13:19:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:12.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:12 smithi152 ceph-mon[34051]: osdmap e367: 8 total, 8 up, 8 in 2024-03-21T13:19:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:19:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:19:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:12 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:13.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:12 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[30095]: osdmap e367: 8 total, 8 up, 8 in 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2467645815' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi089-79447-66"}]': finished 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[39209]: osdmap e367: 8 total, 8 up, 8 in 2024-03-21T13:19:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:19:13.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:19:13.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:13.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:12 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:13 smithi152 ceph-mon[34051]: pgmap v517: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:19:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:13 smithi152 ceph-mon[34051]: osdmap e368: 8 total, 8 up, 8 in 2024-03-21T13:19:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:14.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[30095]: pgmap v517: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[30095]: osdmap e368: 8 total, 8 up, 8 in 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[39209]: pgmap v517: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:19:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[39209]: osdmap e368: 8 total, 8 up, 8 in 2024-03-21T13:19:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]': finished 2024-03-21T13:19:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:14 smithi152 ceph-mon[34051]: osdmap e369: 8 total, 8 up, 8 in 2024-03-21T13:19:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]: dispatch 2024-03-21T13:19:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]: dispatch 2024-03-21T13:19:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:14 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]': finished 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[30095]: osdmap e369: 8 total, 8 up, 8 in 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]: dispatch 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]: dispatch 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-51"}]': finished 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[39209]: osdmap e369: 8 total, 8 up, 8 in 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]: dispatch 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]: dispatch 2024-03-21T13:19:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:14 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:15 smithi152 ceph-mon[34051]: pgmap v520: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:15 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:19:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]': finished 2024-03-21T13:19:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:15 smithi152 ceph-mon[34051]: osdmap e370: 8 total, 8 up, 8 in 2024-03-21T13:19:16.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:19:16.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[39209]: pgmap v520: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]': finished 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[39209]: osdmap e370: 8 total, 8 up, 8 in 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[30095]: pgmap v520: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-51", "mode": "writeback"}]': finished 2024-03-21T13:19:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[30095]: osdmap e370: 8 total, 8 up, 8 in 2024-03-21T13:19:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:19:16.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:19:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:19:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:19:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:17 smithi152 ceph-mon[34051]: osdmap e371: 8 total, 8 up, 8 in 2024-03-21T13:19:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:19:17.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:19:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:19:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[30095]: osdmap e371: 8 total, 8 up, 8 in 2024-03-21T13:19:17.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:19:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:19:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:19:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[39209]: osdmap e371: 8 total, 8 up, 8 in 2024-03-21T13:19:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:19:17.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:19:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:19:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:18.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:19:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:19:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:18 smithi152 ceph-mon[34051]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:19:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:19:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:18 smithi152 ceph-mon[34051]: osdmap e372: 8 total, 8 up, 8 in 2024-03-21T13:19:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:19:18.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[30095]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[30095]: osdmap e372: 8 total, 8 up, 8 in 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[39209]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[39209]: osdmap e372: 8 total, 8 up, 8 in 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:19:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:19:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:19 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:19:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:19:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:19 smithi152 ceph-mon[34051]: osdmap e373: 8 total, 8 up, 8 in 2024-03-21T13:19:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:19:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[30095]: osdmap e373: 8 total, 8 up, 8 in 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:19:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:19:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[39209]: osdmap e373: 8 total, 8 up, 8 in 2024-03-21T13:19:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:19:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:19.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:19:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:20 smithi152 ceph-mon[34051]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:19:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:19:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:19:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:20 smithi152 ceph-mon[34051]: osdmap e374: 8 total, 8 up, 8 in 2024-03-21T13:19:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[30095]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[30095]: osdmap e374: 8 total, 8 up, 8 in 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[39209]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[39209]: osdmap e374: 8 total, 8 up, 8 in 2024-03-21T13:19:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:19:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]': finished 2024-03-21T13:19:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:21 smithi152 ceph-mon[34051]: osdmap e375: 8 total, 8 up, 8 in 2024-03-21T13:19:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]': finished 2024-03-21T13:19:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:21 smithi089 ceph-mon[30095]: osdmap e375: 8 total, 8 up, 8 in 2024-03-21T13:19:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-51","var": "target_max_objects","val": "1"}]': finished 2024-03-21T13:19:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:21 smithi089 ceph-mon[39209]: osdmap e375: 8 total, 8 up, 8 in 2024-03-21T13:19:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:22 smithi152 ceph-mon[34051]: pgmap v529: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:22 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-21T13:19:22.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:22 smithi089 ceph-mon[30095]: pgmap v529: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:22.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:22 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-21T13:19:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:22 smithi089 ceph-mon[39209]: pgmap v529: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:22 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-21T13:19:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:24 smithi152 ceph-mon[34051]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:24 smithi089 ceph-mon[30095]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:24 smithi089 ceph-mon[39209]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:26.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:26 smithi152 ceph-mon[34051]: pgmap v531: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:26.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:26 smithi089 ceph-mon[30095]: pgmap v531: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:26.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:26 smithi089 ceph-mon[39209]: pgmap v531: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:19:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:19:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:28.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:19:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:19:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:28 smithi152 ceph-mon[34051]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:28 smithi089 ceph-mon[30095]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:28 smithi089 ceph-mon[39209]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:29.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:29.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:30 smithi152 ceph-mon[34051]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:30 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:19:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[30095]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:19:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[39209]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:19:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:31 smithi152 ceph-mon[34051]: osdmap e376: 8 total, 8 up, 8 in 2024-03-21T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:31 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:19:31.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:31 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:19:31.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:31 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[39209]: osdmap e376: 8 total, 8 up, 8 in 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:19:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[30095]: osdmap e376: 8 total, 8 up, 8 in 2024-03-21T13:19:31.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:31.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:31.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:19:31.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:19:31.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:31 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:33.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:32 smithi152 ceph-mon[34051]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:33.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]': finished 2024-03-21T13:19:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:32 smithi152 ceph-mon[34051]: osdmap e377: 8 total, 8 up, 8 in 2024-03-21T13:19:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:33.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[30095]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]': finished 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[30095]: osdmap e377: 8 total, 8 up, 8 in 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[39209]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]': finished 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[39209]: osdmap e377: 8 total, 8 up, 8 in 2024-03-21T13:19:33.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:33.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-51"}]: dispatch 2024-03-21T13:19:34.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:33 smithi152 ceph-mon[34051]: osdmap e378: 8 total, 8 up, 8 in 2024-03-21T13:19:34.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:33 smithi152 ceph-mon[34051]: osdmap e379: 8 total, 8 up, 8 in 2024-03-21T13:19:34.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:34.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[30095]: osdmap e378: 8 total, 8 up, 8 in 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[30095]: osdmap e379: 8 total, 8 up, 8 in 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[39209]: osdmap e378: 8 total, 8 up, 8 in 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[39209]: osdmap e379: 8 total, 8 up, 8 in 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:34.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: pgmap v538: 196 pgs: 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T13:19:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-21T13:19:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:35.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: osdmap e380: 8 total, 8 up, 8 in 2024-03-21T13:19:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-53"}]: dispatch 2024-03-21T13:19:35.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-53"}]: dispatch 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: pgmap v538: 196 pgs: 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: osdmap e380: 8 total, 8 up, 8 in 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-53"}]: dispatch 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-53"}]: dispatch 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: pgmap v538: 196 pgs: 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:35.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: osdmap e380: 8 total, 8 up, 8 in 2024-03-21T13:19:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-53"}]: dispatch 2024-03-21T13:19:35.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-53"}]: dispatch 2024-03-21T13:19:36.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:36 smithi152 ceph-mon[34051]: pgmap v541: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:36.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:36 smithi152 ceph-mon[34051]: osdmap e381: 8 total, 8 up, 8 in 2024-03-21T13:19:36.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:36 smithi089 ceph-mon[30095]: pgmap v541: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:36 smithi089 ceph-mon[30095]: osdmap e381: 8 total, 8 up, 8 in 2024-03-21T13:19:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:36 smithi089 ceph-mon[39209]: pgmap v541: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:36 smithi089 ceph-mon[39209]: osdmap e381: 8 total, 8 up, 8 in 2024-03-21T13:19:37.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:37 smithi152 ceph-mon[34051]: osdmap e382: 8 total, 8 up, 8 in 2024-03-21T13:19:37.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:37.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:19:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:19:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:37.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:37 smithi089 ceph-mon[39209]: osdmap e382: 8 total, 8 up, 8 in 2024-03-21T13:19:37.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:37.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:37.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:37 smithi089 ceph-mon[30095]: osdmap e382: 8 total, 8 up, 8 in 2024-03-21T13:19:37.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:37.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:38.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:19:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: pgmap v544: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: osdmap e383: 8 total, 8 up, 8 in 2024-03-21T13:19:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:38.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:38.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-55"}]: dispatch 2024-03-21T13:19:38.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-55"}]: dispatch 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: pgmap v544: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: osdmap e383: 8 total, 8 up, 8 in 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-55"}]: dispatch 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-55"}]: dispatch 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: pgmap v544: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:38.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: osdmap e383: 8 total, 8 up, 8 in 2024-03-21T13:19:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-55"}]: dispatch 2024-03-21T13:19:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-55"}]: dispatch 2024-03-21T13:19:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:39 smithi152 ceph-mon[34051]: osdmap e384: 8 total, 8 up, 8 in 2024-03-21T13:19:39.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:39.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:39 smithi089 ceph-mon[30095]: osdmap e384: 8 total, 8 up, 8 in 2024-03-21T13:19:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:39 smithi089 ceph-mon[39209]: osdmap e384: 8 total, 8 up, 8 in 2024-03-21T13:19:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:40 smithi152 ceph-mon[34051]: pgmap v547: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:40 smithi152 ceph-mon[34051]: osdmap e385: 8 total, 8 up, 8 in 2024-03-21T13:19:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:40.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[39209]: pgmap v547: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:40.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[39209]: osdmap e385: 8 total, 8 up, 8 in 2024-03-21T13:19:40.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:40.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:40.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[30095]: pgmap v547: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:40.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[30095]: osdmap e385: 8 total, 8 up, 8 in 2024-03-21T13:19:40.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:40.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:41 smithi152 ceph-mon[34051]: osdmap e386: 8 total, 8 up, 8 in 2024-03-21T13:19:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-57"}]: dispatch 2024-03-21T13:19:41.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-57"}]: dispatch 2024-03-21T13:19:41.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[30095]: osdmap e386: 8 total, 8 up, 8 in 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-57"}]: dispatch 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-57"}]: dispatch 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[39209]: osdmap e386: 8 total, 8 up, 8 in 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-57"}]: dispatch 2024-03-21T13:19:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-57"}]: dispatch 2024-03-21T13:19:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:42 smithi152 ceph-mon[34051]: pgmap v550: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:19:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:42 smithi152 ceph-mon[34051]: osdmap e387: 8 total, 8 up, 8 in 2024-03-21T13:19:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:42 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[30095]: pgmap v550: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[30095]: osdmap e387: 8 total, 8 up, 8 in 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[39209]: pgmap v550: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[39209]: osdmap e387: 8 total, 8 up, 8 in 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:19:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:43 smithi152 ceph-mon[34051]: osdmap e388: 8 total, 8 up, 8 in 2024-03-21T13:19:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:43 smithi089 ceph-mon[30095]: osdmap e388: 8 total, 8 up, 8 in 2024-03-21T13:19:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:43 smithi089 ceph-mon[39209]: osdmap e388: 8 total, 8 up, 8 in 2024-03-21T13:19:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:19:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:44.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: osdmap e389: 8 total, 8 up, 8 in 2024-03-21T13:19:44.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:44.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:44.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:44.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-59"}]: dispatch 2024-03-21T13:19:44.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-59"}]: dispatch 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: osdmap e389: 8 total, 8 up, 8 in 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-59"}]: dispatch 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-59"}]: dispatch 2024-03-21T13:19:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: osdmap e389: 8 total, 8 up, 8 in 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-59"}]: dispatch 2024-03-21T13:19:44.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-59"}]: dispatch 2024-03-21T13:19:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:45 smithi152 ceph-mon[34051]: osdmap e390: 8 total, 8 up, 8 in 2024-03-21T13:19:45.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:45 smithi089 ceph-mon[30095]: osdmap e390: 8 total, 8 up, 8 in 2024-03-21T13:19:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:45 smithi089 ceph-mon[39209]: osdmap e390: 8 total, 8 up, 8 in 2024-03-21T13:19:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:46 smithi152 ceph-mon[34051]: pgmap v556: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:46 smithi152 ceph-mon[34051]: osdmap e391: 8 total, 8 up, 8 in 2024-03-21T13:19:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[30095]: pgmap v556: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[30095]: osdmap e391: 8 total, 8 up, 8 in 2024-03-21T13:19:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[39209]: pgmap v556: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:19:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[39209]: osdmap e391: 8 total, 8 up, 8 in 2024-03-21T13:19:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:47 smithi152 ceph-mon[34051]: osdmap e392: 8 total, 8 up, 8 in 2024-03-21T13:19:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:19:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[30095]: osdmap e392: 8 total, 8 up, 8 in 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[39209]: osdmap e392: 8 total, 8 up, 8 in 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:48.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:19:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-mon[34051]: pgmap v559: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:19:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:19:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-mon[34051]: osdmap e393: 8 total, 8 up, 8 in 2024-03-21T13:19:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:48.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:48.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-61"}]: dispatch 2024-03-21T13:19:48.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-61"}]: dispatch 2024-03-21T13:19:48.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[30095]: pgmap v559: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[30095]: osdmap e393: 8 total, 8 up, 8 in 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-61"}]: dispatch 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-61"}]: dispatch 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[39209]: pgmap v559: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[39209]: osdmap e393: 8 total, 8 up, 8 in 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:19:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-61"}]: dispatch 2024-03-21T13:19:48.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-61"}]: dispatch 2024-03-21T13:19:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:49 smithi152 ceph-mon[34051]: osdmap e394: 8 total, 8 up, 8 in 2024-03-21T13:19:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:49 smithi089 ceph-mon[30095]: osdmap e394: 8 total, 8 up, 8 in 2024-03-21T13:19:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:19:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:49 smithi089 ceph-mon[39209]: osdmap e394: 8 total, 8 up, 8 in 2024-03-21T13:19:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:50 smithi152 ceph-mon[34051]: pgmap v562: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:19:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:50 smithi152 ceph-mon[34051]: osdmap e395: 8 total, 8 up, 8 in 2024-03-21T13:19:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:51.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:50 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[30095]: pgmap v562: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[30095]: osdmap e395: 8 total, 8 up, 8 in 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[39209]: pgmap v562: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[39209]: osdmap e395: 8 total, 8 up, 8 in 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:51.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:19:51.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:50 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:19:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:51 smithi152 ceph-mon[34051]: osdmap e396: 8 total, 8 up, 8 in 2024-03-21T13:19:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[30095]: osdmap e396: 8 total, 8 up, 8 in 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[39209]: osdmap e396: 8 total, 8 up, 8 in 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:52.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:19:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:52 smithi152 ceph-mon[34051]: pgmap v565: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:19:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:19:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:52 smithi152 ceph-mon[34051]: osdmap e397: 8 total, 8 up, 8 in 2024-03-21T13:19:53.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:52 smithi089 ceph-mon[30095]: pgmap v565: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:19:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:19:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:52 smithi089 ceph-mon[30095]: osdmap e397: 8 total, 8 up, 8 in 2024-03-21T13:19:53.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:52 smithi089 ceph-mon[39209]: pgmap v565: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:19:53.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:19:53.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:52 smithi089 ceph-mon[39209]: osdmap e397: 8 total, 8 up, 8 in 2024-03-21T13:19:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:53 smithi152 ceph-mon[34051]: osdmap e398: 8 total, 8 up, 8 in 2024-03-21T13:19:54.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:53 smithi089 ceph-mon[30095]: osdmap e398: 8 total, 8 up, 8 in 2024-03-21T13:19:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:53 smithi089 ceph-mon[39209]: osdmap e398: 8 total, 8 up, 8 in 2024-03-21T13:19:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:54 smithi152 ceph-mon[34051]: pgmap v568: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:54 smithi152 ceph-mon[34051]: osdmap e399: 8 total, 8 up, 8 in 2024-03-21T13:19:55.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:54 smithi089 ceph-mon[30095]: pgmap v568: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:54 smithi089 ceph-mon[30095]: osdmap e399: 8 total, 8 up, 8 in 2024-03-21T13:19:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:54 smithi089 ceph-mon[39209]: pgmap v568: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:19:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:54 smithi089 ceph-mon[39209]: osdmap e399: 8 total, 8 up, 8 in 2024-03-21T13:19:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: osdmap e400: 8 total, 8 up, 8 in 2024-03-21T13:19:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]: dispatch 2024-03-21T13:19:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]: dispatch 2024-03-21T13:19:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]: dispatch 2024-03-21T13:19:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:55 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]: dispatch 2024-03-21T13:19:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: osdmap e400: 8 total, 8 up, 8 in 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: osdmap e400: 8 total, 8 up, 8 in 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]: dispatch 2024-03-21T13:19:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]: dispatch 2024-03-21T13:19:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]: dispatch 2024-03-21T13:19:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]: dispatch 2024-03-21T13:19:56.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:55 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]: dispatch 2024-03-21T13:19:56.971 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:56 smithi152 ceph-mon[34051]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T13:19:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]': finished 2024-03-21T13:19:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-03-21T13:19:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]': finished 2024-03-21T13:19:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:56 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]': finished 2024-03-21T13:19:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:56 smithi152 ceph-mon[34051]: osdmap e401: 8 total, 8 up, 8 in 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[39209]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]': finished 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]': finished 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]': finished 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[39209]: osdmap e401: 8 total, 8 up, 8 in 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[30095]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.0", "id": [0, 6]}]': finished 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-03-21T13:19:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]': finished 2024-03-21T13:19:57.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1e", "id": [4, 6]}]': finished 2024-03-21T13:19:57.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:56 smithi089 ceph-mon[30095]: osdmap e401: 8 total, 8 up, 8 in 2024-03-21T13:19:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:19:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:19:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:19:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:57 smithi089 ceph-mon[30095]: osdmap e402: 8 total, 8 up, 8 in 2024-03-21T13:19:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:57 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:57 smithi089 ceph-mon[39209]: osdmap e402: 8 total, 8 up, 8 in 2024-03-21T13:19:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:57 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:58.048 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:57 smithi152 ceph-mon[34051]: osdmap e402: 8 total, 8 up, 8 in 2024-03-21T13:19:58.048 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:57 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:19:58.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:19:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:19:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:58 smithi152 ceph-mon[34051]: pgmap v574: 228 pgs: 4 peering, 224 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-21T13:19:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:58 smithi089 ceph-mon[30095]: pgmap v574: 228 pgs: 4 peering, 224 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-21T13:19:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:58 smithi089 ceph-mon[39209]: pgmap v574: 228 pgs: 4 peering, 224 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-21T13:20:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:19:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:00.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:19:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:19:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:00 smithi152 ceph-mon[34051]: pgmap v575: 228 pgs: 4 peering, 224 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-21T13:20:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:00 smithi152 ceph-mon[34051]: osdmap e403: 8 total, 8 up, 8 in 2024-03-21T13:20:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:00 smithi152 ceph-mon[34051]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-21T13:20:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:00 smithi152 ceph-mon[34051]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-21T13:20:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:00 smithi152 ceph-mon[34051]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-21T13:20:01.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:00 smithi152 ceph-mon[34051]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-21T13:20:01.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[30095]: pgmap v575: 228 pgs: 4 peering, 224 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[30095]: osdmap e403: 8 total, 8 up, 8 in 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[30095]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[30095]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[30095]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[30095]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[39209]: pgmap v575: 228 pgs: 4 peering, 224 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[39209]: osdmap e403: 8 total, 8 up, 8 in 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[39209]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[39209]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[39209]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-21T13:20:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:00 smithi089 ceph-mon[39209]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-21T13:20:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:02 smithi152 ceph-mon[34051]: pgmap v577: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T13:20:03.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:02 smithi089 ceph-mon[39209]: pgmap v577: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T13:20:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:02 smithi089 ceph-mon[30095]: pgmap v577: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T13:20:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:03 smithi152 ceph-mon[34051]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T13:20:04.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:03 smithi089 ceph-mon[39209]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T13:20:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:03 smithi089 ceph-mon[30095]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T13:20:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:05 smithi152 ceph-mon[34051]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-03-21T13:20:06.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:05 smithi089 ceph-mon[39209]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-03-21T13:20:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:05 smithi089 ceph-mon[30095]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-03-21T13:20:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:06 smithi152 ceph-mon[34051]: osdmap e404: 8 total, 8 up, 8 in 2024-03-21T13:20:07.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:06 smithi089 ceph-mon[39209]: osdmap e404: 8 total, 8 up, 8 in 2024-03-21T13:20:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:06 smithi089 ceph-mon[30095]: osdmap e404: 8 total, 8 up, 8 in 2024-03-21T13:20:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:20:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:20:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:07 smithi152 ceph-mon[34051]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:20:08.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:20:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:20:08.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:07 smithi089 ceph-mon[30095]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:20:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:07 smithi089 ceph-mon[39209]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:20:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:08 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:09.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:08 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:08 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:10.226 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:09 smithi089 ceph-mon[30095]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:10.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:09 smithi089 ceph-mon[39209]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:09 smithi152 ceph-mon[34051]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:12.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:11 smithi089 ceph-mon[30095]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:12.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:11 smithi089 ceph-mon[39209]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:11 smithi152 ceph-mon[34051]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:13.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:12 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:13.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:12 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:12 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:13 smithi152 ceph-mon[34051]: pgmap v584: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:14.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:13 smithi089 ceph-mon[30095]: pgmap v584: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:13 smithi089 ceph-mon[39209]: pgmap v584: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:16.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:15 smithi152 ceph-mon[34051]: osdmap e405: 8 total, 8 up, 8 in 2024-03-21T13:20:16.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:15 smithi089 ceph-mon[30095]: osdmap e405: 8 total, 8 up, 8 in 2024-03-21T13:20:16.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:15 smithi089 ceph-mon[39209]: osdmap e405: 8 total, 8 up, 8 in 2024-03-21T13:20:17.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:16 smithi152 ceph-mon[34051]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 802 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:17.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:16 smithi089 ceph-mon[30095]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 802 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:17.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:16 smithi089 ceph-mon[39209]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 802 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:17.802 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:20:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:20:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:18.066 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:17 smithi152 ceph-mon[34051]: osdmap e406: 8 total, 8 up, 8 in 2024-03-21T13:20:18.066 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:17 smithi152 ceph-mon[34051]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:18.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:17 smithi089 ceph-mon[30095]: osdmap e406: 8 total, 8 up, 8 in 2024-03-21T13:20:18.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:17 smithi089 ceph-mon[30095]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:18.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:17 smithi089 ceph-mon[39209]: osdmap e406: 8 total, 8 up, 8 in 2024-03-21T13:20:18.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:17 smithi089 ceph-mon[39209]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:20:18.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:20:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:20:19.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:18 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:18 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:19.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:18 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:19 smithi152 ceph-mon[34051]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 0 op/s 2024-03-21T13:20:20.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:19 smithi089 ceph-mon[39209]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 0 op/s 2024-03-21T13:20:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:19 smithi089 ceph-mon[30095]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 0 op/s 2024-03-21T13:20:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:21 smithi152 ceph-mon[34051]: pgmap v590: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:22.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:21 smithi089 ceph-mon[39209]: pgmap v590: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:22.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:21 smithi089 ceph-mon[30095]: pgmap v590: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:23 smithi152 ceph-mon[34051]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:20:24.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:23 smithi089 ceph-mon[39209]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:20:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:23 smithi089 ceph-mon[30095]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:20:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:25 smithi152 ceph-mon[34051]: osdmap e407: 8 total, 8 up, 8 in 2024-03-21T13:20:26.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:25 smithi089 ceph-mon[39209]: osdmap e407: 8 total, 8 up, 8 in 2024-03-21T13:20:26.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:25 smithi089 ceph-mon[30095]: osdmap e407: 8 total, 8 up, 8 in 2024-03-21T13:20:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:26 smithi152 ceph-mon[34051]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-03-21T13:20:27.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:26 smithi089 ceph-mon[39209]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-03-21T13:20:27.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:26 smithi089 ceph-mon[30095]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-03-21T13:20:27.799 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:20:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-63"}]: dispatch 2024-03-21T13:20:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-63"}]: dispatch 2024-03-21T13:20:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:27 smithi152 ceph-mon[34051]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-63"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-63"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[30095]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-63"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-63"}]: dispatch 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[39209]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:20:28.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:28.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:20:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:20:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:28 smithi152 ceph-mon[34051]: osdmap e408: 8 total, 8 up, 8 in 2024-03-21T13:20:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:28 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:28 smithi089 ceph-mon[30095]: osdmap e408: 8 total, 8 up, 8 in 2024-03-21T13:20:29.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:28 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:28 smithi089 ceph-mon[39209]: osdmap e408: 8 total, 8 up, 8 in 2024-03-21T13:20:29.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:28 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:29 smithi152 ceph-mon[34051]: osdmap e409: 8 total, 8 up, 8 in 2024-03-21T13:20:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:29 smithi152 ceph-mon[34051]: pgmap v597: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:30.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[39209]: osdmap e409: 8 total, 8 up, 8 in 2024-03-21T13:20:30.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:30.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:30.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[39209]: pgmap v597: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[30095]: osdmap e409: 8 total, 8 up, 8 in 2024-03-21T13:20:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:30.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:29 smithi089 ceph-mon[30095]: pgmap v597: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[30095]: osdmap e410: 8 total, 8 up, 8 in 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:20:31.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[39209]: osdmap e410: 8 total, 8 up, 8 in 2024-03-21T13:20:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:31.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:30 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:20:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:30 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:30 smithi152 ceph-mon[34051]: osdmap e410: 8 total, 8 up, 8 in 2024-03-21T13:20:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:30 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:20:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:20:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:31 smithi152 ceph-mon[34051]: osdmap e411: 8 total, 8 up, 8 in 2024-03-21T13:20:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:31 smithi152 ceph-mon[34051]: pgmap v600: 228 pgs: 10 unknown, 218 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:31 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:20:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:31 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:20:32.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:31 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:20:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:20:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[39209]: osdmap e411: 8 total, 8 up, 8 in 2024-03-21T13:20:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[39209]: pgmap v600: 228 pgs: 10 unknown, 218 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:20:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:20:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:20:32.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:20:32.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[30095]: osdmap e411: 8 total, 8 up, 8 in 2024-03-21T13:20:32.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[30095]: pgmap v600: 228 pgs: 10 unknown, 218 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:32.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:20:32.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:20:32.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:31 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:20:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:32 smithi152 ceph-mon[34051]: osdmap e412: 8 total, 8 up, 8 in 2024-03-21T13:20:33.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:32 smithi089 ceph-mon[39209]: osdmap e412: 8 total, 8 up, 8 in 2024-03-21T13:20:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:32 smithi089 ceph-mon[30095]: osdmap e412: 8 total, 8 up, 8 in 2024-03-21T13:20:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:33 smithi152 ceph-mon[34051]: osdmap e413: 8 total, 8 up, 8 in 2024-03-21T13:20:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:33 smithi152 ceph-mon[34051]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:20:34.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:33 smithi089 ceph-mon[30095]: osdmap e413: 8 total, 8 up, 8 in 2024-03-21T13:20:34.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:33 smithi089 ceph-mon[30095]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:20:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:33 smithi089 ceph-mon[39209]: osdmap e413: 8 total, 8 up, 8 in 2024-03-21T13:20:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:33 smithi089 ceph-mon[39209]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:20:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:34 smithi152 ceph-mon[34051]: osdmap e414: 8 total, 8 up, 8 in 2024-03-21T13:20:35.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:34 smithi089 ceph-mon[30095]: osdmap e414: 8 total, 8 up, 8 in 2024-03-21T13:20:35.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:34 smithi089 ceph-mon[39209]: osdmap e414: 8 total, 8 up, 8 in 2024-03-21T13:20:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:35 smithi152 ceph-mon[34051]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:20:36.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:35 smithi089 ceph-mon[30095]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:20:36.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:35 smithi089 ceph-mon[39209]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:20:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:20:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:20:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:37 smithi152 ceph-mon[34051]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-21T13:20:38.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:20:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:20:38.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:37 smithi089 ceph-mon[30095]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-21T13:20:38.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:37 smithi089 ceph-mon[39209]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-21T13:20:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:38 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:39.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:38 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:38 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:40.225 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:39 smithi089 ceph-mon[30095]: pgmap v607: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 579 B/s wr, 3 op/s 2024-03-21T13:20:40.226 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:39 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:40.226 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:39 smithi089 ceph-mon[30095]: osdmap e415: 8 total, 8 up, 8 in 2024-03-21T13:20:40.226 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:39 smithi089 ceph-mon[39209]: pgmap v607: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 579 B/s wr, 3 op/s 2024-03-21T13:20:40.226 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:39 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:40.226 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:39 smithi089 ceph-mon[39209]: osdmap e415: 8 total, 8 up, 8 in 2024-03-21T13:20:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:39 smithi152 ceph-mon[34051]: pgmap v607: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 579 B/s wr, 3 op/s 2024-03-21T13:20:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:39 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:39 smithi152 ceph-mon[34051]: osdmap e415: 8 total, 8 up, 8 in 2024-03-21T13:20:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:41 smithi152 ceph-mon[34051]: pgmap v609: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:20:42.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:41 smithi089 ceph-mon[39209]: pgmap v609: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:20:42.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:41 smithi089 ceph-mon[30095]: pgmap v609: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:20:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:43.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:43 smithi152 ceph-mon[34051]: pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 452 B/s wr, 3 op/s 2024-03-21T13:20:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-65"}]: dispatch 2024-03-21T13:20:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-65"}]: dispatch 2024-03-21T13:20:44.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[30095]: pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 452 B/s wr, 3 op/s 2024-03-21T13:20:44.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-65"}]: dispatch 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-65"}]: dispatch 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[39209]: pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 452 B/s wr, 3 op/s 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-65"}]: dispatch 2024-03-21T13:20:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-65"}]: dispatch 2024-03-21T13:20:45.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:44 smithi152 ceph-mon[34051]: osdmap e416: 8 total, 8 up, 8 in 2024-03-21T13:20:45.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:44 smithi089 ceph-mon[30095]: osdmap e416: 8 total, 8 up, 8 in 2024-03-21T13:20:45.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:44 smithi089 ceph-mon[39209]: osdmap e416: 8 total, 8 up, 8 in 2024-03-21T13:20:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:45 smithi152 ceph-mon[34051]: pgmap v612: 196 pgs: 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:45 smithi152 ceph-mon[34051]: osdmap e417: 8 total, 8 up, 8 in 2024-03-21T13:20:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:46.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:46.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[30095]: pgmap v612: 196 pgs: 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:46.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[30095]: osdmap e417: 8 total, 8 up, 8 in 2024-03-21T13:20:46.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:46.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:46.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[39209]: pgmap v612: 196 pgs: 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:46.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[39209]: osdmap e417: 8 total, 8 up, 8 in 2024-03-21T13:20:46.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:46.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:47.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:47.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:47 smithi152 ceph-mon[34051]: osdmap e418: 8 total, 8 up, 8 in 2024-03-21T13:20:47.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:47.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:47.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:47 smithi089 ceph-mon[30095]: osdmap e418: 8 total, 8 up, 8 in 2024-03-21T13:20:47.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:47 smithi089 ceph-mon[39209]: osdmap e418: 8 total, 8 up, 8 in 2024-03-21T13:20:47.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:20:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:20:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:48.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:20:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:20:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:48 smithi152 ceph-mon[34051]: pgmap v615: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:48 smithi152 ceph-mon[34051]: osdmap e419: 8 total, 8 up, 8 in 2024-03-21T13:20:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:48.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-67"}]: dispatch 2024-03-21T13:20:48.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-67"}]: dispatch 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[39209]: pgmap v615: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[39209]: osdmap e419: 8 total, 8 up, 8 in 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-67"}]: dispatch 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-67"}]: dispatch 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[30095]: pgmap v615: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[30095]: osdmap e419: 8 total, 8 up, 8 in 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:48.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-67"}]: dispatch 2024-03-21T13:20:48.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-67"}]: dispatch 2024-03-21T13:20:49.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:49 smithi152 ceph-mon[34051]: osdmap e420: 8 total, 8 up, 8 in 2024-03-21T13:20:49.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:49.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:49 smithi089 ceph-mon[30095]: osdmap e420: 8 total, 8 up, 8 in 2024-03-21T13:20:49.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:49.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:49 smithi089 ceph-mon[39209]: osdmap e420: 8 total, 8 up, 8 in 2024-03-21T13:20:49.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:50.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:50 smithi152 ceph-mon[34051]: pgmap v618: 196 pgs: 196 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:50 smithi152 ceph-mon[34051]: osdmap e421: 8 total, 8 up, 8 in 2024-03-21T13:20:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:50.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[30095]: pgmap v618: 196 pgs: 196 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[30095]: osdmap e421: 8 total, 8 up, 8 in 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[39209]: pgmap v618: 196 pgs: 196 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[39209]: osdmap e421: 8 total, 8 up, 8 in 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:50.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:51 smithi152 ceph-mon[34051]: osdmap e422: 8 total, 8 up, 8 in 2024-03-21T13:20:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-69"}]: dispatch 2024-03-21T13:20:51.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-69"}]: dispatch 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[30095]: osdmap e422: 8 total, 8 up, 8 in 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-69"}]: dispatch 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-69"}]: dispatch 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[39209]: osdmap e422: 8 total, 8 up, 8 in 2024-03-21T13:20:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:20:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-69"}]: dispatch 2024-03-21T13:20:51.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-69"}]: dispatch 2024-03-21T13:20:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:52 smithi152 ceph-mon[34051]: pgmap v621: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-21T13:20:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:52 smithi152 ceph-mon[34051]: osdmap e423: 8 total, 8 up, 8 in 2024-03-21T13:20:52.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:52 smithi089 ceph-mon[30095]: pgmap v621: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-21T13:20:52.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:52 smithi089 ceph-mon[30095]: osdmap e423: 8 total, 8 up, 8 in 2024-03-21T13:20:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:52 smithi089 ceph-mon[39209]: pgmap v621: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-21T13:20:52.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:52 smithi089 ceph-mon[39209]: osdmap e423: 8 total, 8 up, 8 in 2024-03-21T13:20:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:53 smithi152 ceph-mon[34051]: osdmap e424: 8 total, 8 up, 8 in 2024-03-21T13:20:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:53.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:53.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:53 smithi089 ceph-mon[39209]: osdmap e424: 8 total, 8 up, 8 in 2024-03-21T13:20:53.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:53.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:53 smithi089 ceph-mon[30095]: osdmap e424: 8 total, 8 up, 8 in 2024-03-21T13:20:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:53.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:20:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:54 smithi152 ceph-mon[34051]: pgmap v624: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:20:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:54 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:54 smithi152 ceph-mon[34051]: osdmap e425: 8 total, 8 up, 8 in 2024-03-21T13:20:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:54.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[39209]: pgmap v624: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:20:54.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[39209]: osdmap e425: 8 total, 8 up, 8 in 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[30095]: pgmap v624: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[30095]: osdmap e425: 8 total, 8 up, 8 in 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:20:54.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:54.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:20:55.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:20:55.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:55 smithi152 ceph-mon[34051]: osdmap e426: 8 total, 8 up, 8 in 2024-03-21T13:20:55.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:20:55.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:55 smithi089 ceph-mon[30095]: osdmap e426: 8 total, 8 up, 8 in 2024-03-21T13:20:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:20:55.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:55 smithi089 ceph-mon[39209]: osdmap e426: 8 total, 8 up, 8 in 2024-03-21T13:20:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:56 smithi152 ceph-mon[34051]: pgmap v627: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:56 smithi152 ceph-mon[34051]: osdmap e427: 8 total, 8 up, 8 in 2024-03-21T13:20:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:56 smithi089 ceph-mon[30095]: pgmap v627: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:56 smithi089 ceph-mon[30095]: osdmap e427: 8 total, 8 up, 8 in 2024-03-21T13:20:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:56 smithi089 ceph-mon[39209]: pgmap v627: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:20:56.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:56 smithi089 ceph-mon[39209]: osdmap e427: 8 total, 8 up, 8 in 2024-03-21T13:20:57.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:20:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:20:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:20:58.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:20:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:20:58.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:58 smithi152 ceph-mon[34051]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-21T13:20:58.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:58.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:58 smithi089 ceph-mon[39209]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-21T13:20:58.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:58.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:58 smithi089 ceph-mon[30095]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-21T13:20:58.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:20:59.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:20:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:59.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:20:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:20:59.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:20:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:00 smithi152 ceph-mon[34051]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-21T13:21:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:00 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:00.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:00 smithi089 ceph-mon[30095]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-21T13:21:00.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:00 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:00 smithi089 ceph-mon[39209]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-21T13:21:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:00 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:02 smithi152 ceph-mon[34051]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:21:02.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:02 smithi089 ceph-mon[39209]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:21:02.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:02 smithi089 ceph-mon[30095]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-21T13:21:04.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:04 smithi152 ceph-mon[34051]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-21T13:21:04.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:04 smithi089 ceph-mon[30095]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-21T13:21:04.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:04 smithi089 ceph-mon[39209]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-21T13:21:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:06 smithi152 ceph-mon[34051]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-21T13:21:06.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:06 smithi089 ceph-mon[30095]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-21T13:21:06.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:06 smithi089 ceph-mon[39209]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-21T13:21:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:21:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:21:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:08.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:08 smithi152 ceph-mon[34051]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-21T13:21:08.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:21:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:21:08.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:08 smithi089 ceph-mon[30095]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-21T13:21:08.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:08 smithi089 ceph-mon[39209]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-21T13:21:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:09.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:09.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:10.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:10 smithi089 ceph-mon[30095]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:10 smithi089 ceph-mon[39209]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:10 smithi152 ceph-mon[34051]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:12.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:12 smithi152 ceph-mon[34051]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:12.785 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:12 smithi089 ceph-mon[30095]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:12.785 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:12 smithi089 ceph-mon[39209]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:13.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:14 smithi152 ceph-mon[34051]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:14.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:14 smithi089 ceph-mon[30095]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:14.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:14 smithi089 ceph-mon[39209]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:16 smithi152 ceph-mon[34051]: pgmap v638: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:16.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-71"}]: dispatch 2024-03-21T13:21:16.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-71"}]: dispatch 2024-03-21T13:21:16.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[30095]: pgmap v638: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-71"}]: dispatch 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-71"}]: dispatch 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[39209]: pgmap v638: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-71"}]: dispatch 2024-03-21T13:21:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-71"}]: dispatch 2024-03-21T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:17 smithi152 ceph-mon[34051]: osdmap e428: 8 total, 8 up, 8 in 2024-03-21T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:17 smithi152 ceph-mon[34051]: osdmap e429: 8 total, 8 up, 8 in 2024-03-21T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:21:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[30095]: osdmap e428: 8 total, 8 up, 8 in 2024-03-21T13:21:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[30095]: osdmap e429: 8 total, 8 up, 8 in 2024-03-21T13:21:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[39209]: osdmap e428: 8 total, 8 up, 8 in 2024-03-21T13:21:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[39209]: osdmap e429: 8 total, 8 up, 8 in 2024-03-21T13:21:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:18 smithi152 ceph-mon[34051]: pgmap v640: 196 pgs: 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:18 smithi152 ceph-mon[34051]: osdmap e430: 8 total, 8 up, 8 in 2024-03-21T13:21:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:18.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:21:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:21:18.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[30095]: pgmap v640: 196 pgs: 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[30095]: osdmap e430: 8 total, 8 up, 8 in 2024-03-21T13:21:18.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:18.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[39209]: pgmap v640: 196 pgs: 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:18.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:18.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[39209]: osdmap e430: 8 total, 8 up, 8 in 2024-03-21T13:21:18.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:19.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:19.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:19 smithi152 ceph-mon[34051]: osdmap e431: 8 total, 8 up, 8 in 2024-03-21T13:21:19.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:19.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:19 smithi089 ceph-mon[30095]: osdmap e431: 8 total, 8 up, 8 in 2024-03-21T13:21:19.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:19.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:19 smithi089 ceph-mon[39209]: osdmap e431: 8 total, 8 up, 8 in 2024-03-21T13:21:20.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:20 smithi152 ceph-mon[34051]: pgmap v643: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:20.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:20 smithi152 ceph-mon[34051]: osdmap e432: 8 total, 8 up, 8 in 2024-03-21T13:21:20.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:20 smithi089 ceph-mon[30095]: pgmap v643: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:20.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:20 smithi089 ceph-mon[30095]: osdmap e432: 8 total, 8 up, 8 in 2024-03-21T13:21:20.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:20 smithi089 ceph-mon[39209]: pgmap v643: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:20.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:20 smithi089 ceph-mon[39209]: osdmap e432: 8 total, 8 up, 8 in 2024-03-21T13:21:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:22 smithi152 ceph-mon[34051]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T13:21:22.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:22 smithi089 ceph-mon[30095]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T13:21:22.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:22 smithi089 ceph-mon[39209]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T13:21:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:24 smithi152 ceph-mon[34051]: pgmap v647: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:21:24.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:24 smithi089 ceph-mon[30095]: pgmap v647: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:21:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:24 smithi089 ceph-mon[39209]: pgmap v647: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-21T13:21:26.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:26 smithi152 ceph-mon[34051]: pgmap v648: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-21T13:21:26.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:26 smithi089 ceph-mon[30095]: pgmap v648: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-21T13:21:26.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:26 smithi089 ceph-mon[39209]: pgmap v648: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-21T13:21:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:21:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:21:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:28.397 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:21:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:21:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:28 smithi152 ceph-mon[34051]: pgmap v649: 228 pgs: 228 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 895 B/s wr, 93 op/s 2024-03-21T13:21:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:28 smithi089 ceph-mon[30095]: pgmap v649: 228 pgs: 228 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 895 B/s wr, 93 op/s 2024-03-21T13:21:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:28 smithi089 ceph-mon[39209]: pgmap v649: 228 pgs: 228 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 895 B/s wr, 93 op/s 2024-03-21T13:21:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:29.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:29.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:30 smithi152 ceph-mon[34051]: pgmap v650: 228 pgs: 228 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 731 B/s wr, 76 op/s 2024-03-21T13:21:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-73"}]: dispatch 2024-03-21T13:21:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-73"}]: dispatch 2024-03-21T13:21:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[30095]: pgmap v650: 228 pgs: 228 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 731 B/s wr, 76 op/s 2024-03-21T13:21:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-73"}]: dispatch 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-73"}]: dispatch 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[39209]: pgmap v650: 228 pgs: 228 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 731 B/s wr, 76 op/s 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-73"}]: dispatch 2024-03-21T13:21:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-73"}]: dispatch 2024-03-21T13:21:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:31 smithi152 ceph-mon[34051]: osdmap e433: 8 total, 8 up, 8 in 2024-03-21T13:21:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:31 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:21:31.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:31 smithi089 ceph-mon[30095]: osdmap e433: 8 total, 8 up, 8 in 2024-03-21T13:21:31.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:31 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:21:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:31 smithi089 ceph-mon[39209]: osdmap e433: 8 total, 8 up, 8 in 2024-03-21T13:21:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:31 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:21:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:32 smithi152 ceph-mon[34051]: pgmap v652: 196 pgs: 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-03-21T13:21:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:32 smithi152 ceph-mon[34051]: osdmap e434: 8 total, 8 up, 8 in 2024-03-21T13:21:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:32 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:32 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:32 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:21:32.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[30095]: pgmap v652: 196 pgs: 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-03-21T13:21:32.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[30095]: osdmap e434: 8 total, 8 up, 8 in 2024-03-21T13:21:32.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:32.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:32.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:21:32.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:21:32.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:21:32.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[39209]: pgmap v652: 196 pgs: 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-03-21T13:21:32.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[39209]: osdmap e434: 8 total, 8 up, 8 in 2024-03-21T13:21:32.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:32.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:32.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:21:32.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:21:32.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:32 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:21:33.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:33.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:33 smithi152 ceph-mon[34051]: osdmap e435: 8 total, 8 up, 8 in 2024-03-21T13:21:33.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:33.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:33.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:33 smithi089 ceph-mon[30095]: osdmap e435: 8 total, 8 up, 8 in 2024-03-21T13:21:33.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:33 smithi089 ceph-mon[39209]: osdmap e435: 8 total, 8 up, 8 in 2024-03-21T13:21:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:34 smithi152 ceph-mon[34051]: pgmap v655: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:34 smithi152 ceph-mon[34051]: osdmap e436: 8 total, 8 up, 8 in 2024-03-21T13:21:34.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:34 smithi089 ceph-mon[30095]: pgmap v655: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:34.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:34 smithi089 ceph-mon[30095]: osdmap e436: 8 total, 8 up, 8 in 2024-03-21T13:21:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:34 smithi089 ceph-mon[39209]: pgmap v655: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:34 smithi089 ceph-mon[39209]: osdmap e436: 8 total, 8 up, 8 in 2024-03-21T13:21:35.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:35 smithi152 ceph-mon[34051]: osdmap e437: 8 total, 8 up, 8 in 2024-03-21T13:21:35.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:35.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:35.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-75"}]: dispatch 2024-03-21T13:21:35.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-75"}]: dispatch 2024-03-21T13:21:35.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[30095]: osdmap e437: 8 total, 8 up, 8 in 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-75"}]: dispatch 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-75"}]: dispatch 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[39209]: osdmap e437: 8 total, 8 up, 8 in 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-75"}]: dispatch 2024-03-21T13:21:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-75"}]: dispatch 2024-03-21T13:21:36.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:36 smithi152 ceph-mon[34051]: pgmap v658: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:36.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:36 smithi152 ceph-mon[34051]: osdmap e438: 8 total, 8 up, 8 in 2024-03-21T13:21:36.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:36 smithi089 ceph-mon[39209]: pgmap v658: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:36 smithi089 ceph-mon[39209]: osdmap e438: 8 total, 8 up, 8 in 2024-03-21T13:21:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:36 smithi089 ceph-mon[30095]: pgmap v658: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:21:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:36 smithi089 ceph-mon[30095]: osdmap e438: 8 total, 8 up, 8 in 2024-03-21T13:21:37.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:37 smithi152 ceph-mon[34051]: osdmap e439: 8 total, 8 up, 8 in 2024-03-21T13:21:37.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:37.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:21:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:21:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:37.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:37 smithi089 ceph-mon[30095]: osdmap e439: 8 total, 8 up, 8 in 2024-03-21T13:21:37.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:37.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:37.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:37 smithi089 ceph-mon[39209]: osdmap e439: 8 total, 8 up, 8 in 2024-03-21T13:21:37.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:37.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:38.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:21:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:21:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:38 smithi152 ceph-mon[34051]: pgmap v661: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:21:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:38 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:38 smithi152 ceph-mon[34051]: osdmap e440: 8 total, 8 up, 8 in 2024-03-21T13:21:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:38.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[39209]: pgmap v661: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:21:38.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[39209]: osdmap e440: 8 total, 8 up, 8 in 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[30095]: pgmap v661: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[30095]: osdmap e440: 8 total, 8 up, 8 in 2024-03-21T13:21:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:39 smithi152 ceph-mon[34051]: osdmap e441: 8 total, 8 up, 8 in 2024-03-21T13:21:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-77"}]: dispatch 2024-03-21T13:21:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-77"}]: dispatch 2024-03-21T13:21:39.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[39209]: osdmap e441: 8 total, 8 up, 8 in 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-77"}]: dispatch 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-77"}]: dispatch 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[30095]: osdmap e441: 8 total, 8 up, 8 in 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:39.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:21:39.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-77"}]: dispatch 2024-03-21T13:21:39.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-77"}]: dispatch 2024-03-21T13:21:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:40 smithi152 ceph-mon[34051]: pgmap v664: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:21:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:40 smithi152 ceph-mon[34051]: osdmap e442: 8 total, 8 up, 8 in 2024-03-21T13:21:40.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:40 smithi089 ceph-mon[30095]: pgmap v664: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:21:40.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:40 smithi089 ceph-mon[30095]: osdmap e442: 8 total, 8 up, 8 in 2024-03-21T13:21:40.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:40 smithi089 ceph-mon[39209]: pgmap v664: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:21:40.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:40 smithi089 ceph-mon[39209]: osdmap e442: 8 total, 8 up, 8 in 2024-03-21T13:21:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:41 smithi152 ceph-mon[34051]: osdmap e443: 8 total, 8 up, 8 in 2024-03-21T13:21:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:41.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:41 smithi152 ceph-mon[34051]: osdmap e444: 8 total, 8 up, 8 in 2024-03-21T13:21:41.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[30095]: osdmap e443: 8 total, 8 up, 8 in 2024-03-21T13:21:41.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[30095]: osdmap e444: 8 total, 8 up, 8 in 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[39209]: osdmap e443: 8 total, 8 up, 8 in 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:21:41.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:41 smithi089 ceph-mon[39209]: osdmap e444: 8 total, 8 up, 8 in 2024-03-21T13:21:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:42 smithi152 ceph-mon[34051]: pgmap v667: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:21:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:42 smithi152 ceph-mon[34051]: osdmap e445: 8 total, 8 up, 8 in 2024-03-21T13:21:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:42 smithi089 ceph-mon[30095]: pgmap v667: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:21:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:42 smithi089 ceph-mon[30095]: osdmap e445: 8 total, 8 up, 8 in 2024-03-21T13:21:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:42 smithi089 ceph-mon[39209]: pgmap v667: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:21:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:21:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:42 smithi089 ceph-mon[39209]: osdmap e445: 8 total, 8 up, 8 in 2024-03-21T13:21:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:43 smithi152 ceph-mon[34051]: osdmap e446: 8 total, 8 up, 8 in 2024-03-21T13:21:43.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:43.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:43 smithi089 ceph-mon[39209]: osdmap e446: 8 total, 8 up, 8 in 2024-03-21T13:21:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:43 smithi089 ceph-mon[30095]: osdmap e446: 8 total, 8 up, 8 in 2024-03-21T13:21:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:44 smithi152 ceph-mon[34051]: pgmap v670: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:21:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:44 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-03-21T13:21:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:44 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-03-21T13:21:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:44 smithi152 ceph-mon[34051]: 305.8 deep-scrub starts 2024-03-21T13:21:44.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:44 smithi152 ceph-mon[34051]: 305.8 deep-scrub ok 2024-03-21T13:21:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[30095]: pgmap v670: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:21:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-03-21T13:21:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-03-21T13:21:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[30095]: 305.8 deep-scrub starts 2024-03-21T13:21:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[30095]: 305.8 deep-scrub ok 2024-03-21T13:21:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[39209]: pgmap v670: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:21:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:21:44.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-03-21T13:21:44.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-03-21T13:21:44.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[39209]: 305.8 deep-scrub starts 2024-03-21T13:21:44.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:44 smithi089 ceph-mon[39209]: 305.8 deep-scrub ok 2024-03-21T13:21:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:46 smithi152 ceph-mon[34051]: pgmap v672: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 462 B/s wr, 2 op/s 2024-03-21T13:21:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:46 smithi089 ceph-mon[30095]: pgmap v672: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 462 B/s wr, 2 op/s 2024-03-21T13:21:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:46 smithi089 ceph-mon[39209]: pgmap v672: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 462 B/s wr, 2 op/s 2024-03-21T13:21:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:21:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:21:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:48.528 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:21:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:21:48.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:48 smithi152 ceph-mon[34051]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-03-21T13:21:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:48 smithi089 ceph-mon[30095]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-03-21T13:21:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:48 smithi089 ceph-mon[39209]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-03-21T13:21:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:49.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:49.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:50 smithi152 ceph-mon[34051]: pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 689 B/s wr, 2 op/s 2024-03-21T13:21:51.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:50 smithi089 ceph-mon[30095]: pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 689 B/s wr, 2 op/s 2024-03-21T13:21:51.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:50 smithi089 ceph-mon[39209]: pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 689 B/s wr, 2 op/s 2024-03-21T13:21:52.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:51 smithi152 ceph-mon[34051]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 608 B/s wr, 2 op/s 2024-03-21T13:21:52.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:51 smithi089 ceph-mon[30095]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 608 B/s wr, 2 op/s 2024-03-21T13:21:52.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:51 smithi089 ceph-mon[39209]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 608 B/s wr, 2 op/s 2024-03-21T13:21:54.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:53 smithi152 ceph-mon[34051]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:21:54.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:53 smithi089 ceph-mon[30095]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:21:54.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:53 smithi089 ceph-mon[39209]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:21:56.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:55 smithi152 ceph-mon[34051]: pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 448 B/s wr, 1 op/s 2024-03-21T13:21:56.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:55 smithi089 ceph-mon[39209]: pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 448 B/s wr, 1 op/s 2024-03-21T13:21:56.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:55 smithi089 ceph-mon[30095]: pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 448 B/s wr, 1 op/s 2024-03-21T13:21:57.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:21:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:21:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:21:58.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:57 smithi152 ceph-mon[34051]: pgmap v678: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-21T13:21:58.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:57 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:58.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:21:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:21:58.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:57 smithi089 ceph-mon[39209]: pgmap v678: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-21T13:21:58.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:57 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:58.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:57 smithi089 ceph-mon[30095]: pgmap v678: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-21T13:21:58.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:57 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:21:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:58 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:59.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:58 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:21:59.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:58 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:21:59 smithi152 ceph-mon[34051]: pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:22:00.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:21:59 smithi089 ceph-mon[30095]: pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:22:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:21:59 smithi089 ceph-mon[39209]: pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:22:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:01 smithi152 ceph-mon[34051]: pgmap v680: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:22:02.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:01 smithi089 ceph-mon[30095]: pgmap v680: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:22:02.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:01 smithi089 ceph-mon[39209]: pgmap v680: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:22:03.821 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:22:03 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:22:03.545+0000 7f9dc224f700 -1 snap_mapper.add_oid found existing snaps mapped on 305:178263fb:test-rados-api-smithi089-79633-80::foo:2, removing 2024-03-21T13:22:03.825 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0[43618]: 2024-03-21T13:22:03.545+0000 7fd8c8520700 -1 snap_mapper.add_oid found existing snaps mapped on 305:178263fb:test-rados-api-smithi089-79633-80::foo:2, removing 2024-03-21T13:22:03.825 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1[49745]: 2024-03-21T13:22:03.545+0000 7f7a927c0700 -1 snap_mapper.add_oid found existing snaps mapped on 305:178263fb:test-rados-api-smithi089-79633-80::foo:2, removing 2024-03-21T13:22:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:03 smithi152 ceph-mon[34051]: pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:22:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-79"}]: dispatch 2024-03-21T13:22:04.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-79"}]: dispatch 2024-03-21T13:22:04.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[30095]: pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:22:04.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:04.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-79"}]: dispatch 2024-03-21T13:22:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-79"}]: dispatch 2024-03-21T13:22:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[39209]: pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:22:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-79"}]: dispatch 2024-03-21T13:22:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-79"}]: dispatch 2024-03-21T13:22:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:05 smithi152 ceph-mon[34051]: osdmap e447: 8 total, 8 up, 8 in 2024-03-21T13:22:05.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:05 smithi089 ceph-mon[30095]: osdmap e447: 8 total, 8 up, 8 in 2024-03-21T13:22:05.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:05 smithi089 ceph-mon[39209]: osdmap e447: 8 total, 8 up, 8 in 2024-03-21T13:22:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:06 smithi152 ceph-mon[34051]: pgmap v683: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:06 smithi152 ceph-mon[34051]: osdmap e448: 8 total, 8 up, 8 in 2024-03-21T13:22:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:06.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[30095]: pgmap v683: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[30095]: osdmap e448: 8 total, 8 up, 8 in 2024-03-21T13:22:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[39209]: pgmap v683: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:22:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[39209]: osdmap e448: 8 total, 8 up, 8 in 2024-03-21T13:22:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:07 smithi152 ceph-mon[34051]: osdmap e449: 8 total, 8 up, 8 in 2024-03-21T13:22:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[30095]: osdmap e449: 8 total, 8 up, 8 in 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[39209]: osdmap e449: 8 total, 8 up, 8 in 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:22:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:22:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:08.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:22:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:22:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:08 smithi152 ceph-mon[34051]: pgmap v686: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:08 smithi152 ceph-mon[34051]: osdmap e450: 8 total, 8 up, 8 in 2024-03-21T13:22:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:08.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[30095]: pgmap v686: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:08.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[30095]: osdmap e450: 8 total, 8 up, 8 in 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[39209]: pgmap v686: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[39209]: osdmap e450: 8 total, 8 up, 8 in 2024-03-21T13:22:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: osdmap e451: 8 total, 8 up, 8 in 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: osdmap e452: 8 total, 8 up, 8 in 2024-03-21T13:22:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:09.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: osdmap e451: 8 total, 8 up, 8 in 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: osdmap e452: 8 total, 8 up, 8 in 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: osdmap e451: 8 total, 8 up, 8 in 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:09.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: osdmap e452: 8 total, 8 up, 8 in 2024-03-21T13:22:09.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:10 smithi152 ceph-mon[34051]: pgmap v689: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:22:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:10 smithi152 ceph-mon[34051]: osdmap e453: 8 total, 8 up, 8 in 2024-03-21T13:22:10.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:10 smithi089 ceph-mon[39209]: pgmap v689: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:22:10.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:10.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:10 smithi089 ceph-mon[39209]: osdmap e453: 8 total, 8 up, 8 in 2024-03-21T13:22:10.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:10 smithi089 ceph-mon[30095]: pgmap v689: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-21T13:22:10.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:10.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:10 smithi089 ceph-mon[30095]: osdmap e453: 8 total, 8 up, 8 in 2024-03-21T13:22:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-21T13:22:11.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-21T13:22:11.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-21T13:22:11.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-21T13:22:11.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-21T13:22:11.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-21T13:22:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:12 smithi152 ceph-mon[34051]: pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-21T13:22:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-21T13:22:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:12 smithi152 ceph-mon[34051]: osdmap e454: 8 total, 8 up, 8 in 2024-03-21T13:22:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-21T13:22:12.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-21T13:22:12.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[30095]: pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[30095]: osdmap e454: 8 total, 8 up, 8 in 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[39209]: pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[39209]: osdmap e454: 8 total, 8 up, 8 in 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-21T13:22:12.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[30095]: osdmap e455: 8 total, 8 up, 8 in 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[39209]: osdmap e455: 8 total, 8 up, 8 in 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:13.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:13.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-21T13:22:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:13 smithi152 ceph-mon[34051]: osdmap e455: 8 total, 8 up, 8 in 2024-03-21T13:22:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:14.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[30095]: pgmap v695: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-21T13:22:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[30095]: osdmap e456: 8 total, 8 up, 8 in 2024-03-21T13:22:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-81"}]: dispatch 2024-03-21T13:22:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-81"}]: dispatch 2024-03-21T13:22:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[39209]: pgmap v695: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-21T13:22:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[39209]: osdmap e456: 8 total, 8 up, 8 in 2024-03-21T13:22:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-81"}]: dispatch 2024-03-21T13:22:14.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-81"}]: dispatch 2024-03-21T13:22:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:14 smithi152 ceph-mon[34051]: pgmap v695: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-21T13:22:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:14 smithi152 ceph-mon[34051]: osdmap e456: 8 total, 8 up, 8 in 2024-03-21T13:22:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:14.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:14.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-81"}]: dispatch 2024-03-21T13:22:14.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-81"}]: dispatch 2024-03-21T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:15 smithi152 ceph-mon[34051]: osdmap e457: 8 total, 8 up, 8 in 2024-03-21T13:22:15.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:15 smithi089 ceph-mon[30095]: osdmap e457: 8 total, 8 up, 8 in 2024-03-21T13:22:15.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:15 smithi089 ceph-mon[39209]: osdmap e457: 8 total, 8 up, 8 in 2024-03-21T13:22:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:16 smithi152 ceph-mon[34051]: pgmap v698: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:16 smithi152 ceph-mon[34051]: osdmap e458: 8 total, 8 up, 8 in 2024-03-21T13:22:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:16.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[30095]: pgmap v698: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[30095]: osdmap e458: 8 total, 8 up, 8 in 2024-03-21T13:22:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:16.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:16.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[39209]: pgmap v698: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:16.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[39209]: osdmap e458: 8 total, 8 up, 8 in 2024-03-21T13:22:16.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:16.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[39209]: osdmap e459: 8 total, 8 up, 8 in 2024-03-21T13:22:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[30095]: osdmap e459: 8 total, 8 up, 8 in 2024-03-21T13:22:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:17.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:17 smithi152 ceph-mon[34051]: osdmap e459: 8 total, 8 up, 8 in 2024-03-21T13:22:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:17.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:22:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:22:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:18.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:22:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:22:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:18 smithi152 ceph-mon[34051]: pgmap v701: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-21T13:22:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:18 smithi152 ceph-mon[34051]: osdmap e460: 8 total, 8 up, 8 in 2024-03-21T13:22:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:18.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[30095]: pgmap v701: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[30095]: osdmap e460: 8 total, 8 up, 8 in 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[39209]: pgmap v701: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[39209]: osdmap e460: 8 total, 8 up, 8 in 2024-03-21T13:22:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:19 smithi152 ceph-mon[34051]: osdmap e461: 8 total, 8 up, 8 in 2024-03-21T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:19.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[30095]: osdmap e461: 8 total, 8 up, 8 in 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[39209]: osdmap e461: 8 total, 8 up, 8 in 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:19.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:20 smithi152 ceph-mon[34051]: pgmap v704: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-21T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:20 smithi152 ceph-mon[34051]: osdmap e462: 8 total, 8 up, 8 in 2024-03-21T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:20.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[30095]: pgmap v704: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[30095]: osdmap e462: 8 total, 8 up, 8 in 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[39209]: pgmap v704: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[39209]: osdmap e462: 8 total, 8 up, 8 in 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:21.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:21.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:21 smithi152 ceph-mon[34051]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T13:22:21.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:21.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:21 smithi089 ceph-mon[30095]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T13:22:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:21 smithi089 ceph-mon[39209]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T13:22:22.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:22 smithi152 ceph-mon[34051]: pgmap v707: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:22.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:22 smithi152 ceph-mon[34051]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T13:22:22.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:22 smithi089 ceph-mon[39209]: pgmap v707: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:22.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:22 smithi089 ceph-mon[39209]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T13:22:22.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:22 smithi089 ceph-mon[30095]: pgmap v707: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:22.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:22 smithi089 ceph-mon[30095]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T13:22:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:23 smithi152 ceph-mon[34051]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T13:22:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-83"}]: dispatch 2024-03-21T13:22:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-83"}]: dispatch 2024-03-21T13:22:23.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[39209]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T13:22:23.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-83"}]: dispatch 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-83"}]: dispatch 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[30095]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-83"}]: dispatch 2024-03-21T13:22:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-83"}]: dispatch 2024-03-21T13:22:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:24 smithi152 ceph-mon[34051]: pgmap v710: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:24 smithi152 ceph-mon[34051]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T13:22:24.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:24 smithi089 ceph-mon[30095]: pgmap v710: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:24.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:24 smithi089 ceph-mon[30095]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T13:22:24.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:24 smithi089 ceph-mon[39209]: pgmap v710: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:24.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:24 smithi089 ceph-mon[39209]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T13:22:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:25 smithi152 ceph-mon[34051]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T13:22:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:25 smithi152 ceph-mon[34051]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T13:22:25.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[30095]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T13:22:25.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[30095]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[39209]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:25 smithi089 ceph-mon[39209]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T13:22:26.472 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:26 smithi152 ceph-mon[34051]: pgmap v713: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:26 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[39209]: pgmap v713: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[30095]: pgmap v713: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:26.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:27.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:27.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[30095]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T13:22:27.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:27.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:27.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:27.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[39209]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T13:22:27.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:27.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:27 smithi152 ceph-mon[34051]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T13:22:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_tier","val": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:22:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:22:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:22:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:28.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:22:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-21T13:22:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T13:22:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T13:22:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:28.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T13:22:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T13:22:28.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:29 smithi152 ceph-mon[34051]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T13:22:29.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:29 smithi089 ceph-mon[30095]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T13:22:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:29 smithi089 ceph-mon[39209]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T13:22:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:30 smithi152 ceph-mon[34051]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-21T13:22:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:30 smithi152 ceph-mon[34051]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T13:22:30.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:30 smithi089 ceph-mon[30095]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-21T13:22:30.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:30 smithi089 ceph-mon[30095]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T13:22:30.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:30 smithi089 ceph-mon[39209]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-21T13:22:30.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:30 smithi089 ceph-mon[39209]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T13:22:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:32 smithi152 ceph-mon[34051]: pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-21T13:22:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:32 smithi152 ceph-mon[34051]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T13:22:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-85"}]: dispatch 2024-03-21T13:22:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-85"}]: dispatch 2024-03-21T13:22:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:32 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:32.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[30095]: pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[30095]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-85"}]: dispatch 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-85"}]: dispatch 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[39209]: pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[39209]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-85"}]: dispatch 2024-03-21T13:22:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-85"}]: dispatch 2024-03-21T13:22:32.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:32 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:22:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T13:22:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi089 to 3248M 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: Adjusting osd_memory_target on smithi152 to 3248M 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:33.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:33 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi089 to 3248M 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: Adjusting osd_memory_target on smithi152 to 3248M 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:33.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi089 to 3248M 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: Adjusting osd_memory_target on smithi152 to 3248M 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:22:33.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:33 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:22:34.249 INFO:tasks.workunit.client.0.smithi089.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18058 ms) 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22487 ms) 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3008 ms) 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3018 ms) 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-21T13:22:34.250 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3027 ms) 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3031 ms) 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4030 ms) 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39244 ms) 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-21T13:22:34.251 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16171 ms) 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4037 ms) 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3030 ms) 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25077 ms) 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.252 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14298 ms) 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5041 ms) 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4033 ms) 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: waiting for scrubs... 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: done waiting 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24512 ms) 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-21T13:22:34.253 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10085 ms) 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9067 ms) 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9060 ms) 2024-03-21T13:22:34.254 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-21T13:22:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:34 smithi152 ceph-mon[34051]: pgmap v725: 196 pgs: 196 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:22:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:34 smithi152 ceph-mon[34051]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T13:22:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[30095]: pgmap v725: 196 pgs: 196 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:22:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[30095]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T13:22:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[39209]: pgmap v725: 196 pgs: 196 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:22:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[39209]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T13:22:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:35 smithi152 ceph-mon[34051]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T13:22:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:35 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[30095]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[39209]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:35 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:36 smithi152 ceph-mon[34051]: pgmap v728: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:36 smithi152 ceph-mon[34051]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T13:22:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[30095]: pgmap v728: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[30095]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[39209]: pgmap v728: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[39209]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]: dispatch 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]: dispatch 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]': finished 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:37.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:37.574 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:22:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]: dispatch 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]: dispatch 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]': finished 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:37.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]: dispatch 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]: dispatch 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_tier","val": "test-rados-api-smithi089-79633-89-test-flush"}]': finished 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:37.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:22:38.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:22:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:22:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:38 smithi152 ceph-mon[34051]: pgmap v731: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-21T13:22:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:38 smithi152 ceph-mon[34051]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T13:22:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[30095]: pgmap v731: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-21T13:22:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[30095]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T13:22:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[39209]: pgmap v731: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-21T13:22:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:22:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[39209]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T13:22:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:22:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:39 smithi152 ceph-mon[34051]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T13:22:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:39 smithi089 ceph-mon[30095]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T13:22:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:22:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:39 smithi089 ceph-mon[39209]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T13:22:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:40 smithi152 ceph-mon[34051]: pgmap v734: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-21T13:22:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:40 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:40 smithi152 ceph-mon[34051]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T13:22:40.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:40 smithi089 ceph-mon[30095]: pgmap v734: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-21T13:22:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:40 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:40 smithi089 ceph-mon[30095]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T13:22:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:40 smithi089 ceph-mon[39209]: pgmap v734: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-21T13:22:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:40 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:40 smithi089 ceph-mon[39209]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T13:22:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:41.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-87"}]: dispatch 2024-03-21T13:22:41.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-87"}]: dispatch 2024-03-21T13:22:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-87"}]: dispatch 2024-03-21T13:22:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-87"}]: dispatch 2024-03-21T13:22:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-87"}]: dispatch 2024-03-21T13:22:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-87"}]: dispatch 2024-03-21T13:22:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:42 smithi152 ceph-mon[34051]: pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:42 smithi152 ceph-mon[34051]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T13:22:42.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:42 smithi089 ceph-mon[30095]: pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:42.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:42 smithi089 ceph-mon[30095]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T13:22:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:42 smithi089 ceph-mon[39209]: pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-21T13:22:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:42 smithi089 ceph-mon[39209]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T13:22:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:43 smithi152 ceph-mon[34051]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T13:22:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:43.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[30095]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T13:22:43.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:43.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:43.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:43.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[39209]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T13:22:43.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:43.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:43.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:44 smithi152 ceph-mon[34051]: pgmap v740: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:22:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:44 smithi152 ceph-mon[34051]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T13:22:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:44.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[30095]: pgmap v740: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[30095]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[39209]: pgmap v740: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[39209]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:44.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:22:45.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:45.723 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:45 smithi152 ceph-mon[34051]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T13:22:45.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:45.724 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:45 smithi089 ceph-mon[30095]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T13:22:45.724 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:22:45.724 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:45 smithi089 ceph-mon[39209]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T13:22:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:46 smithi152 ceph-mon[34051]: pgmap v743: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:46 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:46 smithi152 ceph-mon[34051]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T13:22:46.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:46 smithi089 ceph-mon[39209]: pgmap v743: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:46.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:46 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:46.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:46 smithi089 ceph-mon[39209]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T13:22:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:46 smithi089 ceph-mon[30095]: pgmap v743: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:46 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:46 smithi089 ceph-mon[30095]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T13:22:47.574 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:22:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[30095]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-90"}]: dispatch 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-90"}]: dispatch 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[39209]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:47.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-90"}]: dispatch 2024-03-21T13:22:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-90"}]: dispatch 2024-03-21T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:47 smithi152 ceph-mon[34051]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-90"}]: dispatch 2024-03-21T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-90"}]: dispatch 2024-03-21T13:22:48.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:48 smithi152 ceph-mon[34051]: pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-21T13:22:48.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:48 smithi152 ceph-mon[34051]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T13:22:48.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:22:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:22:48.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:48 smithi089 ceph-mon[39209]: pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-21T13:22:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:48 smithi089 ceph-mon[39209]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T13:22:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:48 smithi089 ceph-mon[30095]: pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-21T13:22:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:48 smithi089 ceph-mon[30095]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:49 smithi152 ceph-mon[34051]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:49.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[39209]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T13:22:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[30095]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T13:22:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:50.321 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:22:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=cleanup t=2024-03-21T13:22:49.916682581Z level=info msg="Completed cleanup jobs" duration=3.495641ms 2024-03-21T13:22:50.321 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:22:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=sqlstore.transactions t=2024-03-21T13:22:49.924148793Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T13:22:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:50 smithi152 ceph-mon[34051]: pgmap v749: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:22:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:50 smithi152 ceph-mon[34051]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T13:22:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:50.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[39209]: pgmap v749: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:22:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[39209]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T13:22:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[30095]: pgmap v749: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:22:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[30095]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T13:22:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:51.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:51 smithi152 ceph-mon[34051]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T13:22:51.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:51 smithi089 ceph-mon[30095]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T13:22:51.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:51 smithi089 ceph-mon[39209]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:52 smithi152 ceph-mon[34051]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:52 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:52 smithi152 ceph-mon[34051]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:52.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:52.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-92"}]: dispatch 2024-03-21T13:22:52.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-92"}]: dispatch 2024-03-21T13:22:52.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[30095]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[30095]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-92"}]: dispatch 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-92"}]: dispatch 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[39209]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[39209]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T13:22:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:52.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:22:52.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-92"}]: dispatch 2024-03-21T13:22:52.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-92"}]: dispatch 2024-03-21T13:22:53.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:53 smithi152 ceph-mon[34051]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T13:22:53.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:53 smithi089 ceph-mon[30095]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T13:22:53.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:53 smithi089 ceph-mon[39209]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:54 smithi152 ceph-mon[34051]: pgmap v755: 196 pgs: 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:54 smithi152 ceph-mon[34051]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[30095]: pgmap v755: 196 pgs: 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[30095]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[39209]: pgmap v755: 196 pgs: 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[39209]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:55 smithi152 ceph-mon[34051]: osdmap e497: 8 total, 8 up, 8 in 2024-03-21T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:55.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:55.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:55 smithi089 ceph-mon[30095]: osdmap e497: 8 total, 8 up, 8 in 2024-03-21T13:22:55.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:55.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:22:55.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:55 smithi089 ceph-mon[39209]: osdmap e497: 8 total, 8 up, 8 in 2024-03-21T13:22:55.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:22:56.722 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:56 smithi152 ceph-mon[34051]: pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:56.722 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:56 smithi152 ceph-mon[34051]: osdmap e498: 8 total, 8 up, 8 in 2024-03-21T13:22:56.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:56 smithi089 ceph-mon[39209]: pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:56.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:56 smithi089 ceph-mon[39209]: osdmap e498: 8 total, 8 up, 8 in 2024-03-21T13:22:56.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:56 smithi089 ceph-mon[30095]: pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:22:56.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:56 smithi089 ceph-mon[30095]: osdmap e498: 8 total, 8 up, 8 in 2024-03-21T13:22:57.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:57 smithi152 ceph-mon[34051]: osdmap e499: 8 total, 8 up, 8 in 2024-03-21T13:22:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:22:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:22:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:22:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:57 smithi089 ceph-mon[30095]: osdmap e499: 8 total, 8 up, 8 in 2024-03-21T13:22:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:57 smithi089 ceph-mon[39209]: osdmap e499: 8 total, 8 up, 8 in 2024-03-21T13:22:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:58 smithi152 ceph-mon[34051]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-21T13:22:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:58.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:22:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:22:58.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:58 smithi089 ceph-mon[30095]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-21T13:22:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:58 smithi089 ceph-mon[39209]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-21T13:22:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:22:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:22:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:59.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:22:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:22:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:22:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:00 smithi152 ceph-mon[34051]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-21T13:23:00.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:00 smithi089 ceph-mon[30095]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-21T13:23:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:00 smithi089 ceph-mon[39209]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-21T13:23:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:02 smithi152 ceph-mon[34051]: pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T13:23:02.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:02 smithi089 ceph-mon[30095]: pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T13:23:02.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:02 smithi089 ceph-mon[39209]: pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T13:23:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:04 smithi152 ceph-mon[34051]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-21T13:23:04.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:04 smithi089 ceph-mon[30095]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-21T13:23:04.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:04 smithi089 ceph-mon[39209]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-21T13:23:06.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:06 smithi152 ceph-mon[34051]: pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 964 B/s wr, 4 op/s 2024-03-21T13:23:06.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:06 smithi089 ceph-mon[30095]: pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 964 B/s wr, 4 op/s 2024-03-21T13:23:06.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:06 smithi089 ceph-mon[39209]: pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 964 B/s wr, 4 op/s 2024-03-21T13:23:07.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:07 smithi152 ceph-mon[34051]: osdmap e500: 8 total, 8 up, 8 in 2024-03-21T13:23:07.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:07 smithi089 ceph-mon[30095]: osdmap e500: 8 total, 8 up, 8 in 2024-03-21T13:23:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:23:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:23:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:23:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:07 smithi089 ceph-mon[39209]: osdmap e500: 8 total, 8 up, 8 in 2024-03-21T13:23:08.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:08 smithi152 ceph-mon[34051]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-21T13:23:08.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:23:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:23:08.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:08 smithi089 ceph-mon[30095]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-21T13:23:08.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:08 smithi089 ceph-mon[39209]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-21T13:23:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:09.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:09.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:10.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:10 smithi089 ceph-mon[30095]: pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-21T13:23:10.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:10 smithi089 ceph-mon[39209]: pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-21T13:23:10.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:10 smithi152 ceph-mon[34051]: pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-21T13:23:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:12 smithi152 ceph-mon[34051]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:12.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:12 smithi089 ceph-mon[30095]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:12.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:12 smithi089 ceph-mon[39209]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:13.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:13.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:13.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:14.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:14 smithi152 ceph-mon[34051]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:14.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:14 smithi089 ceph-mon[30095]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:14.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:14 smithi089 ceph-mon[39209]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:15 smithi152 ceph-mon[34051]: osdmap e501: 8 total, 8 up, 8 in 2024-03-21T13:23:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:15 smithi152 ceph-mon[34051]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:16.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:15 smithi089 ceph-mon[30095]: osdmap e501: 8 total, 8 up, 8 in 2024-03-21T13:23:16.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:15 smithi089 ceph-mon[30095]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:16.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:15 smithi089 ceph-mon[39209]: osdmap e501: 8 total, 8 up, 8 in 2024-03-21T13:23:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:15 smithi089 ceph-mon[39209]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:17.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:17.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:17.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-94"}]: dispatch 2024-03-21T13:23:17.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-94"}]: dispatch 2024-03-21T13:23:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:17.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-94"}]: dispatch 2024-03-21T13:23:17.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-94"}]: dispatch 2024-03-21T13:23:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-94"}]: dispatch 2024-03-21T13:23:17.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-94"}]: dispatch 2024-03-21T13:23:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:23:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:23:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:23:18.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:23:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:23:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:18 smithi152 ceph-mon[34051]: osdmap e502: 8 total, 8 up, 8 in 2024-03-21T13:23:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:18 smithi152 ceph-mon[34051]: pgmap v774: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:23:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:18 smithi152 ceph-mon[34051]: osdmap e503: 8 total, 8 up, 8 in 2024-03-21T13:23:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:18.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:18.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[30095]: osdmap e502: 8 total, 8 up, 8 in 2024-03-21T13:23:18.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[30095]: pgmap v774: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:23:18.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[30095]: osdmap e503: 8 total, 8 up, 8 in 2024-03-21T13:23:18.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[39209]: osdmap e502: 8 total, 8 up, 8 in 2024-03-21T13:23:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[39209]: pgmap v774: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:23:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[39209]: osdmap e503: 8 total, 8 up, 8 in 2024-03-21T13:23:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:18.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:19 smithi152 ceph-mon[34051]: osdmap e504: 8 total, 8 up, 8 in 2024-03-21T13:23:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:23:19.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:19.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[30095]: osdmap e504: 8 total, 8 up, 8 in 2024-03-21T13:23:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:23:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[39209]: osdmap e504: 8 total, 8 up, 8 in 2024-03-21T13:23:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:23:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:20 smithi152 ceph-mon[34051]: pgmap v777: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:20 smithi152 ceph-mon[34051]: osdmap e505: 8 total, 8 up, 8 in 2024-03-21T13:23:20.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:20 smithi089 ceph-mon[39209]: pgmap v777: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:20.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:20 smithi089 ceph-mon[39209]: osdmap e505: 8 total, 8 up, 8 in 2024-03-21T13:23:20.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:20 smithi089 ceph-mon[30095]: pgmap v777: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:20.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:20 smithi089 ceph-mon[30095]: osdmap e505: 8 total, 8 up, 8 in 2024-03-21T13:23:22.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:22 smithi152 ceph-mon[34051]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:23:22.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:22 smithi089 ceph-mon[30095]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:23:22.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:22 smithi089 ceph-mon[39209]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T13:23:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:24 smithi152 ceph-mon[34051]: pgmap v780: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:23:24.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:24 smithi089 ceph-mon[39209]: pgmap v780: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:23:24.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:24 smithi089 ceph-mon[30095]: pgmap v780: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 2 op/s 2024-03-21T13:23:26.373 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:26 smithi152 ceph-mon[34051]: pgmap v781: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 719 B/s rd, 863 B/s wr, 2 op/s 2024-03-21T13:23:26.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:26 smithi089 ceph-mon[30095]: pgmap v781: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 719 B/s rd, 863 B/s wr, 2 op/s 2024-03-21T13:23:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:26 smithi089 ceph-mon[39209]: pgmap v781: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 719 B/s rd, 863 B/s wr, 2 op/s 2024-03-21T13:23:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:23:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:23:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:23:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:28 smithi152 ceph-mon[34051]: pgmap v782: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 757 B/s wr, 2 op/s 2024-03-21T13:23:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:28.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:23:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:23:28.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:28 smithi089 ceph-mon[30095]: pgmap v782: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 757 B/s wr, 2 op/s 2024-03-21T13:23:28.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:28.578 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:28 smithi089 ceph-mon[39209]: pgmap v782: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 757 B/s wr, 2 op/s 2024-03-21T13:23:28.579 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:29.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:30.128 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:23:29 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:23:29.877+0000 7f2b07aac700 -1 snap_mapper.add_oid found existing snaps mapped on 116:1e09a9f3:test-rados-api-smithi089-79633-97::foo:21, removing 2024-03-21T13:23:30.128 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:23:29 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3[62387]: 2024-03-21T13:23:29.877+0000 7fc57bae7700 -1 snap_mapper.add_oid found existing snaps mapped on 116:1e09a9f3:test-rados-api-smithi089-79633-97::foo:21, removing 2024-03-21T13:23:30.129 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:23:29 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0[43618]: 2024-03-21T13:23:29.877+0000 7fd8cc528700 -1 snap_mapper.add_oid found existing snaps mapped on 116:1e09a9f3:test-rados-api-smithi089-79633-97::foo:21, removing 2024-03-21T13:23:30.129 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[30095]: pgmap v783: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-03-21T13:23:30.129 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:30.129 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:30.129 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-96"}]: dispatch 2024-03-21T13:23:30.129 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-96"}]: dispatch 2024-03-21T13:23:30.129 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[39209]: pgmap v783: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-03-21T13:23:30.130 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:30.130 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:30.130 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-96"}]: dispatch 2024-03-21T13:23:30.130 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-96"}]: dispatch 2024-03-21T13:23:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:30 smithi152 ceph-mon[34051]: pgmap v783: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-03-21T13:23:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-96"}]: dispatch 2024-03-21T13:23:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-96"}]: dispatch 2024-03-21T13:23:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:31 smithi152 ceph-mon[34051]: osdmap e506: 8 total, 8 up, 8 in 2024-03-21T13:23:31.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:31 smithi089 ceph-mon[39209]: osdmap e506: 8 total, 8 up, 8 in 2024-03-21T13:23:31.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:31 smithi089 ceph-mon[30095]: osdmap e506: 8 total, 8 up, 8 in 2024-03-21T13:23:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:32 smithi152 ceph-mon[34051]: pgmap v785: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-03-21T13:23:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:32 smithi152 ceph-mon[34051]: osdmap e507: 8 total, 8 up, 8 in 2024-03-21T13:23:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:32.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[30095]: pgmap v785: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-03-21T13:23:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[30095]: osdmap e507: 8 total, 8 up, 8 in 2024-03-21T13:23:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[39209]: pgmap v785: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-03-21T13:23:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[39209]: osdmap e507: 8 total, 8 up, 8 in 2024-03-21T13:23:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:33 smithi152 ceph-mon[34051]: osdmap e508: 8 total, 8 up, 8 in 2024-03-21T13:23:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:33 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:23:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[30095]: osdmap e508: 8 total, 8 up, 8 in 2024-03-21T13:23:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:23:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[39209]: osdmap e508: 8 total, 8 up, 8 in 2024-03-21T13:23:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:33 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:23:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: pgmap v788: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:23:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:23:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]: dispatch 2024-03-21T13:23:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: osdmap e509: 8 total, 8 up, 8 in 2024-03-21T13:23:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]: dispatch 2024-03-21T13:23:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:23:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:23:34.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:34 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:23:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: pgmap v788: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:23:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:23:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]: dispatch 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: osdmap e509: 8 total, 8 up, 8 in 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]: dispatch 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: pgmap v788: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]: dispatch 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: osdmap e509: 8 total, 8 up, 8 in 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]: dispatch 2024-03-21T13:23:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:23:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:23:34.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:34 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:23:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:35 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:23:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]': finished 2024-03-21T13:23:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:35 smithi152 ceph-mon[34051]: osdmap e510: 8 total, 8 up, 8 in 2024-03-21T13:23:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:35.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:23:35.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]': finished 2024-03-21T13:23:35.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[30095]: osdmap e510: 8 total, 8 up, 8 in 2024-03-21T13:23:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:23:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-98", "mode": "writeback"}]': finished 2024-03-21T13:23:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[39209]: osdmap e510: 8 total, 8 up, 8 in 2024-03-21T13:23:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:36 smithi152 ceph-mon[34051]: pgmap v791: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]': finished 2024-03-21T13:23:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:23:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:36 smithi152 ceph-mon[34051]: osdmap e511: 8 total, 8 up, 8 in 2024-03-21T13:23:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:23:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[30095]: pgmap v791: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]': finished 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[30095]: osdmap e511: 8 total, 8 up, 8 in 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[39209]: pgmap v791: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-98"}]': finished 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[39209]: osdmap e511: 8 total, 8 up, 8 in 2024-03-21T13:23:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:23:37.495 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[30095]: osdmap e512: 8 total, 8 up, 8 in 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-21T13:23:37.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[39209]: osdmap e512: 8 total, 8 up, 8 in 2024-03-21T13:23:37.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-21T13:23:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:37 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:23:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:23:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-21T13:23:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:37 smithi152 ceph-mon[34051]: osdmap e512: 8 total, 8 up, 8 in 2024-03-21T13:23:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:37 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-21T13:23:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:23:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:23:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:23:38.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:23:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:23:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:38 smithi152 ceph-mon[34051]: pgmap v794: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:23:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]': finished 2024-03-21T13:23:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:38 smithi152 ceph-mon[34051]: osdmap e513: 8 total, 8 up, 8 in 2024-03-21T13:23:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[30095]: pgmap v794: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:23:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]': finished 2024-03-21T13:23:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[30095]: osdmap e513: 8 total, 8 up, 8 in 2024-03-21T13:23:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[39209]: pgmap v794: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:23:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_count","val": "1"}]': finished 2024-03-21T13:23:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[39209]: osdmap e513: 8 total, 8 up, 8 in 2024-03-21T13:23:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:23:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:39 smithi152 ceph-mon[34051]: osdmap e514: 8 total, 8 up, 8 in 2024-03-21T13:23:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-21T13:23:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-21T13:23:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]': finished 2024-03-21T13:23:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:23:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[30095]: osdmap e514: 8 total, 8 up, 8 in 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]': finished 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[39209]: osdmap e514: 8 total, 8 up, 8 in 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-98","var": "target_max_objects","val": "250"}]': finished 2024-03-21T13:23:40.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[30095]: pgmap v797: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:23:40.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[30095]: osdmap e515: 8 total, 8 up, 8 in 2024-03-21T13:23:40.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:40.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[39209]: pgmap v797: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:23:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[39209]: osdmap e515: 8 total, 8 up, 8 in 2024-03-21T13:23:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:40 smithi152 ceph-mon[34051]: pgmap v797: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:23:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:40 smithi152 ceph-mon[34051]: osdmap e515: 8 total, 8 up, 8 in 2024-03-21T13:23:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:23:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:41 smithi152 ceph-mon[34051]: osdmap e516: 8 total, 8 up, 8 in 2024-03-21T13:23:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:41.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:23:41.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[30095]: osdmap e516: 8 total, 8 up, 8 in 2024-03-21T13:23:41.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:23:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[39209]: osdmap e516: 8 total, 8 up, 8 in 2024-03-21T13:23:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]: dispatch 2024-03-21T13:23:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:42 smithi152 ceph-mon[34051]: pgmap v800: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]': finished 2024-03-21T13:23:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:42 smithi152 ceph-mon[34051]: osdmap e517: 8 total, 8 up, 8 in 2024-03-21T13:23:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:42 smithi089 ceph-mon[30095]: pgmap v800: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]': finished 2024-03-21T13:23:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:42 smithi089 ceph-mon[30095]: osdmap e517: 8 total, 8 up, 8 in 2024-03-21T13:23:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:42 smithi089 ceph-mon[39209]: pgmap v800: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-98"}]': finished 2024-03-21T13:23:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:42 smithi089 ceph-mon[39209]: osdmap e517: 8 total, 8 up, 8 in 2024-03-21T13:23:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:43 smithi152 ceph-mon[34051]: osdmap e518: 8 total, 8 up, 8 in 2024-03-21T13:23:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:43 smithi152 ceph-mon[34051]: osdmap e519: 8 total, 8 up, 8 in 2024-03-21T13:23:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:43.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[30095]: osdmap e518: 8 total, 8 up, 8 in 2024-03-21T13:23:43.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[30095]: osdmap e519: 8 total, 8 up, 8 in 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[39209]: osdmap e518: 8 total, 8 up, 8 in 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[39209]: osdmap e519: 8 total, 8 up, 8 in 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:44 smithi152 ceph-mon[34051]: pgmap v803: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:44 smithi152 ceph-mon[34051]: osdmap e520: 8 total, 8 up, 8 in 2024-03-21T13:23:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[30095]: pgmap v803: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[30095]: osdmap e520: 8 total, 8 up, 8 in 2024-03-21T13:23:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[39209]: pgmap v803: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[39209]: osdmap e520: 8 total, 8 up, 8 in 2024-03-21T13:23:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:23:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:46 smithi152 ceph-mon[34051]: pgmap v806: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:23:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:46 smithi152 ceph-mon[34051]: osdmap e521: 8 total, 8 up, 8 in 2024-03-21T13:23:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:46.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[39209]: pgmap v806: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:46.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:23:46.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[39209]: osdmap e521: 8 total, 8 up, 8 in 2024-03-21T13:23:46.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[30095]: pgmap v806: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:23:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[30095]: osdmap e521: 8 total, 8 up, 8 in 2024-03-21T13:23:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]': finished 2024-03-21T13:23:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]: dispatch 2024-03-21T13:23:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: osdmap e522: 8 total, 8 up, 8 in 2024-03-21T13:23:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]: dispatch 2024-03-21T13:23:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:23:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]': finished 2024-03-21T13:23:47.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: osdmap e523: 8 total, 8 up, 8 in 2024-03-21T13:23:47.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:23:47.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:23:47.574 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:23:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]': finished 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]: dispatch 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: osdmap e522: 8 total, 8 up, 8 in 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]: dispatch 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]': finished 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: osdmap e523: 8 total, 8 up, 8 in 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-6", "overlaypool": "test-rados-api-smithi089-79633-100"}]': finished 2024-03-21T13:23:47.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]: dispatch 2024-03-21T13:23:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: osdmap e522: 8 total, 8 up, 8 in 2024-03-21T13:23:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]: dispatch 2024-03-21T13:23:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:23:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-100", "mode": "writeback"}]': finished 2024-03-21T13:23:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: osdmap e523: 8 total, 8 up, 8 in 2024-03-21T13:23:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:23:47.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:23:48.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:23:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:23:48.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:48 smithi152 ceph-mon[34051]: pgmap v809: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:48.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:23:48.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:48 smithi152 ceph-mon[34051]: osdmap e524: 8 total, 8 up, 8 in 2024-03-21T13:23:48.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:48.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:48.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[30095]: pgmap v809: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:48.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:23:48.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[30095]: osdmap e524: 8 total, 8 up, 8 in 2024-03-21T13:23:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[39209]: pgmap v809: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:23:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[39209]: osdmap e524: 8 total, 8 up, 8 in 2024-03-21T13:23:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:23:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:23:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:49 smithi152 ceph-mon[34051]: osdmap e525: 8 total, 8 up, 8 in 2024-03-21T13:23:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:23:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:23:49.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:49.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:23:49.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[30095]: osdmap e525: 8 total, 8 up, 8 in 2024-03-21T13:23:49.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:23:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:23:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:23:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[39209]: osdmap e525: 8 total, 8 up, 8 in 2024-03-21T13:23:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:23:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:23:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:50 smithi152 ceph-mon[34051]: pgmap v812: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:50 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:23:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:23:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-21T13:23:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:50 smithi152 ceph-mon[34051]: osdmap e526: 8 total, 8 up, 8 in 2024-03-21T13:23:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-21T13:23:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[30095]: pgmap v812: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:23:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[30095]: osdmap e526: 8 total, 8 up, 8 in 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[39209]: pgmap v812: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[39209]: osdmap e526: 8 total, 8 up, 8 in 2024-03-21T13:23:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-21T13:23:52.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[30095]: pgmap v815: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:52.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-21T13:23:52.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[30095]: osdmap e527: 8 total, 8 up, 8 in 2024-03-21T13:23:52.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:52.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:52 smithi152 ceph-mon[34051]: pgmap v815: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:52.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-21T13:23:52.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:52 smithi152 ceph-mon[34051]: osdmap e527: 8 total, 8 up, 8 in 2024-03-21T13:23:52.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:52.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:52.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[39209]: pgmap v815: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-21T13:23:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[39209]: osdmap e527: 8 total, 8 up, 8 in 2024-03-21T13:23:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:23:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:23:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:53 smithi152 ceph-mon[34051]: osdmap e528: 8 total, 8 up, 8 in 2024-03-21T13:23:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:23:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[30095]: osdmap e528: 8 total, 8 up, 8 in 2024-03-21T13:23:53.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:53.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]': finished 2024-03-21T13:23:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[39209]: osdmap e528: 8 total, 8 up, 8 in 2024-03-21T13:23:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]: dispatch 2024-03-21T13:23:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:54 smithi152 ceph-mon[34051]: pgmap v818: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]': finished 2024-03-21T13:23:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:54 smithi152 ceph-mon[34051]: osdmap e529: 8 total, 8 up, 8 in 2024-03-21T13:23:54.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:54 smithi089 ceph-mon[30095]: pgmap v818: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]': finished 2024-03-21T13:23:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:54 smithi089 ceph-mon[30095]: osdmap e529: 8 total, 8 up, 8 in 2024-03-21T13:23:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:54 smithi089 ceph-mon[39209]: pgmap v818: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:23:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-100"}]': finished 2024-03-21T13:23:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:54 smithi089 ceph-mon[39209]: osdmap e529: 8 total, 8 up, 8 in 2024-03-21T13:23:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:55 smithi152 ceph-mon[34051]: osdmap e530: 8 total, 8 up, 8 in 2024-03-21T13:23:55.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:55 smithi089 ceph-mon[30095]: osdmap e530: 8 total, 8 up, 8 in 2024-03-21T13:23:55.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:55 smithi089 ceph-mon[39209]: osdmap e530: 8 total, 8 up, 8 in 2024-03-21T13:23:56.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:56 smithi152 ceph-mon[34051]: pgmap v821: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:56.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:56 smithi152 ceph-mon[34051]: osdmap e531: 8 total, 8 up, 8 in 2024-03-21T13:23:56.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:56.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:56.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[30095]: pgmap v821: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:56.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[30095]: osdmap e531: 8 total, 8 up, 8 in 2024-03-21T13:23:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[39209]: pgmap v821: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:23:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[39209]: osdmap e531: 8 total, 8 up, 8 in 2024-03-21T13:23:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:23:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:57 smithi152 ceph-mon[34051]: osdmap e532: 8 total, 8 up, 8 in 2024-03-21T13:23:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:23:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:23:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:23:57.574 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:23:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[39209]: osdmap e532: 8 total, 8 up, 8 in 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[30095]: osdmap e532: 8 total, 8 up, 8 in 2024-03-21T13:23:57.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-21T13:23:57.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:23:57.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-21T13:23:58.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:23:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:23:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:58 smithi152 ceph-mon[34051]: pgmap v824: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:23:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:58 smithi152 ceph-mon[34051]: osdmap e533: 8 total, 8 up, 8 in 2024-03-21T13:23:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:23:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:23:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:58.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[30095]: pgmap v824: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:58.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:23:58.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[30095]: osdmap e533: 8 total, 8 up, 8 in 2024-03-21T13:23:58.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[39209]: pgmap v824: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[39209]: osdmap e533: 8 total, 8 up, 8 in 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-21T13:23:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[39209]: osdmap e534: 8 total, 8 up, 8 in 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[39209]: osdmap e535: 8 total, 8 up, 8 in 2024-03-21T13:23:59.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:59.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:23:59.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[30095]: osdmap e534: 8 total, 8 up, 8 in 2024-03-21T13:23:59.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:23:59.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:23:59.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:23:59.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:23:59 smithi089 ceph-mon[30095]: osdmap e535: 8 total, 8 up, 8 in 2024-03-21T13:23:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:23:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-21T13:23:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:59 smithi152 ceph-mon[34051]: osdmap e534: 8 total, 8 up, 8 in 2024-03-21T13:23:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:23:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-21T13:23:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-21T13:23:59.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:23:59 smithi152 ceph-mon[34051]: osdmap e535: 8 total, 8 up, 8 in 2024-03-21T13:24:00.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[39209]: pgmap v827: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-102"}]: dispatch 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-102"}]: dispatch 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[30095]: pgmap v827: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-102"}]: dispatch 2024-03-21T13:24:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-102"}]: dispatch 2024-03-21T13:24:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:00 smithi152 ceph-mon[34051]: pgmap v827: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:24:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-6"}]: dispatch 2024-03-21T13:24:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1743343034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-102"}]: dispatch 2024-03-21T13:24:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-6", "tierpool": "test-rados-api-smithi089-79633-102"}]: dispatch 2024-03-21T13:24:01.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:01 smithi152 ceph-mon[34051]: osdmap e536: 8 total, 8 up, 8 in 2024-03-21T13:24:01.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:01 smithi089 ceph-mon[30095]: osdmap e536: 8 total, 8 up, 8 in 2024-03-21T13:24:01.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:01 smithi089 ceph-mon[39209]: osdmap e536: 8 total, 8 up, 8 in 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: pgmap v830: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: osdmap e537: 8 total, 8 up, 8 in 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: pgmap v830: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: osdmap e537: 8 total, 8 up, 8 in 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: pgmap v830: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:24:02.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: osdmap e537: 8 total, 8 up, 8 in 2024-03-21T13:24:02.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:02.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:02.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:03 smithi152 ceph-mon[34051]: osdmap e538: 8 total, 8 up, 8 in 2024-03-21T13:24:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:04.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:04.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[30095]: osdmap e538: 8 total, 8 up, 8 in 2024-03-21T13:24:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:04.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[39209]: osdmap e538: 8 total, 8 up, 8 in 2024-03-21T13:24:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:04 smithi152 ceph-mon[34051]: pgmap v833: 164 pgs: 164 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:04 smithi152 ceph-mon[34051]: osdmap e539: 8 total, 8 up, 8 in 2024-03-21T13:24:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:04 smithi152 ceph-mon[34051]: osdmap e540: 8 total, 8 up, 8 in 2024-03-21T13:24:05.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[30095]: pgmap v833: 164 pgs: 164 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:05.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[30095]: osdmap e539: 8 total, 8 up, 8 in 2024-03-21T13:24:05.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:05.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[30095]: osdmap e540: 8 total, 8 up, 8 in 2024-03-21T13:24:05.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[39209]: pgmap v833: 164 pgs: 164 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:05.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[39209]: osdmap e539: 8 total, 8 up, 8 in 2024-03-21T13:24:05.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi089-79633-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:05.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:04 smithi089 ceph-mon[39209]: osdmap e540: 8 total, 8 up, 8 in 2024-03-21T13:24:06.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:06 smithi152 ceph-mon[34051]: pgmap v836: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:06.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:06 smithi152 ceph-mon[34051]: osdmap e541: 8 total, 8 up, 8 in 2024-03-21T13:24:06.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:06 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:06.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:06 smithi089 ceph-mon[30095]: pgmap v836: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:06.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:06 smithi089 ceph-mon[30095]: osdmap e541: 8 total, 8 up, 8 in 2024-03-21T13:24:06.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:06 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:06.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:06 smithi089 ceph-mon[39209]: pgmap v836: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:06.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:06 smithi089 ceph-mon[39209]: osdmap e541: 8 total, 8 up, 8 in 2024-03-21T13:24:06.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:06 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:07.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:07 smithi152 ceph-mon[34051]: osdmap e542: 8 total, 8 up, 8 in 2024-03-21T13:24:07.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:24:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:24:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:24:07.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:07 smithi089 ceph-mon[30095]: osdmap e542: 8 total, 8 up, 8 in 2024-03-21T13:24:07.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:07 smithi089 ceph-mon[39209]: osdmap e542: 8 total, 8 up, 8 in 2024-03-21T13:24:07.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:08.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:08 smithi152 ceph-mon[34051]: pgmap v839: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:08.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:08.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:08 smithi152 ceph-mon[34051]: osdmap e543: 8 total, 8 up, 8 in 2024-03-21T13:24:08.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:24:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:24:08.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:08 smithi089 ceph-mon[30095]: pgmap v839: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:08.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:08.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:08 smithi089 ceph-mon[30095]: osdmap e543: 8 total, 8 up, 8 in 2024-03-21T13:24:08.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:08 smithi089 ceph-mon[39209]: pgmap v839: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:08.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:08.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:08 smithi089 ceph-mon[39209]: osdmap e543: 8 total, 8 up, 8 in 2024-03-21T13:24:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:09 smithi152 ceph-mon[34051]: osdmap e544: 8 total, 8 up, 8 in 2024-03-21T13:24:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:09 smithi152 ceph-mon[34051]: osdmap e545: 8 total, 8 up, 8 in 2024-03-21T13:24:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:09 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:09.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:09.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[30095]: osdmap e544: 8 total, 8 up, 8 in 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[30095]: osdmap e545: 8 total, 8 up, 8 in 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[39209]: osdmap e544: 8 total, 8 up, 8 in 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[39209]: osdmap e545: 8 total, 8 up, 8 in 2024-03-21T13:24:09.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:09 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:10.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[30095]: pgmap v842: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:10.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:10.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[30095]: osdmap e546: 8 total, 8 up, 8 in 2024-03-21T13:24:10.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-107", "overlaypool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[39209]: pgmap v842: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[39209]: osdmap e546: 8 total, 8 up, 8 in 2024-03-21T13:24:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-107", "overlaypool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:10.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:10 smithi152 ceph-mon[34051]: pgmap v842: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:24:10.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:10.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:10 smithi152 ceph-mon[34051]: osdmap e546: 8 total, 8 up, 8 in 2024-03-21T13:24:10.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-107", "overlaypool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:12 smithi152 ceph-mon[34051]: pgmap v845: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-107", "overlaypool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:12 smithi152 ceph-mon[34051]: osdmap e547: 8 total, 8 up, 8 in 2024-03-21T13:24:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-107-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:24:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:12 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:12.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[30095]: pgmap v845: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:12.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-107", "overlaypool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:12.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[30095]: osdmap e547: 8 total, 8 up, 8 in 2024-03-21T13:24:12.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-107-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:24:12.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:12.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[39209]: pgmap v845: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:12.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-107", "overlaypool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:12.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[39209]: osdmap e547: 8 total, 8 up, 8 in 2024-03-21T13:24:12.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-107-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:24:12.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:12 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:13 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-107-cache", "mode": "writeback"}]': finished 2024-03-21T13:24:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:13 smithi152 ceph-mon[34051]: osdmap e548: 8 total, 8 up, 8 in 2024-03-21T13:24:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-107"}]: dispatch 2024-03-21T13:24:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:13.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:13.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-107-cache", "mode": "writeback"}]': finished 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[39209]: osdmap e548: 8 total, 8 up, 8 in 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-107"}]: dispatch 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-107-cache", "mode": "writeback"}]': finished 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[30095]: osdmap e548: 8 total, 8 up, 8 in 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-107"}]: dispatch 2024-03-21T13:24:13.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:13.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:14.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:14 smithi152 ceph-mon[34051]: pgmap v848: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:14.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-107"}]': finished 2024-03-21T13:24:14.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:14 smithi152 ceph-mon[34051]: osdmap e549: 8 total, 8 up, 8 in 2024-03-21T13:24:14.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:14.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[30095]: pgmap v848: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:14.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-107"}]': finished 2024-03-21T13:24:14.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[30095]: osdmap e549: 8 total, 8 up, 8 in 2024-03-21T13:24:14.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:14.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[39209]: pgmap v848: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:14.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-107"}]': finished 2024-03-21T13:24:14.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[39209]: osdmap e549: 8 total, 8 up, 8 in 2024-03-21T13:24:14.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]: dispatch 2024-03-21T13:24:15.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:15 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:15.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:15.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:15 smithi152 ceph-mon[34051]: osdmap e550: 8 total, 8 up, 8 in 2024-03-21T13:24:15.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:15 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:15.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:15.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:15 smithi089 ceph-mon[39209]: osdmap e550: 8 total, 8 up, 8 in 2024-03-21T13:24:15.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:15 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:15.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4190573874' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-107", "tierpool": "test-rados-api-smithi089-79633-107-cache"}]': finished 2024-03-21T13:24:15.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:15 smithi089 ceph-mon[30095]: osdmap e550: 8 total, 8 up, 8 in 2024-03-21T13:24:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:16 smithi152 ceph-mon[34051]: pgmap v851: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:16 smithi152 ceph-mon[34051]: osdmap e551: 8 total, 8 up, 8 in 2024-03-21T13:24:16.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:16 smithi089 ceph-mon[30095]: pgmap v851: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:16.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:16 smithi089 ceph-mon[30095]: osdmap e551: 8 total, 8 up, 8 in 2024-03-21T13:24:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:16 smithi089 ceph-mon[39209]: pgmap v851: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:16.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:16 smithi089 ceph-mon[39209]: osdmap e551: 8 total, 8 up, 8 in 2024-03-21T13:24:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:17 smithi152 ceph-mon[34051]: osdmap e552: 8 total, 8 up, 8 in 2024-03-21T13:24:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:17.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:24:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:24:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[30095]: osdmap e552: 8 total, 8 up, 8 in 2024-03-21T13:24:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:17.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[39209]: osdmap e552: 8 total, 8 up, 8 in 2024-03-21T13:24:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:17.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:17.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:18.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:24:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:24:19.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:18 smithi152 ceph-mon[34051]: pgmap v854: 172 pgs: 172 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:19.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:19.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:18 smithi152 ceph-mon[34051]: osdmap e553: 8 total, 8 up, 8 in 2024-03-21T13:24:19.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:19.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[39209]: pgmap v854: 172 pgs: 172 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:19.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:19.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[39209]: osdmap e553: 8 total, 8 up, 8 in 2024-03-21T13:24:19.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[30095]: pgmap v854: 172 pgs: 172 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[30095]: osdmap e553: 8 total, 8 up, 8 in 2024-03-21T13:24:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:19 smithi152 ceph-mon[34051]: osdmap e554: 8 total, 8 up, 8 in 2024-03-21T13:24:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:19 smithi152 ceph-mon[34051]: osdmap e555: 8 total, 8 up, 8 in 2024-03-21T13:24:20.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:20.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[39209]: osdmap e554: 8 total, 8 up, 8 in 2024-03-21T13:24:20.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:20.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[39209]: osdmap e555: 8 total, 8 up, 8 in 2024-03-21T13:24:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[30095]: osdmap e554: 8 total, 8 up, 8 in 2024-03-21T13:24:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:19 smithi089 ceph-mon[30095]: osdmap e555: 8 total, 8 up, 8 in 2024-03-21T13:24:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:20 smithi152 ceph-mon[34051]: pgmap v857: 172 pgs: 172 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:20 smithi152 ceph-mon[34051]: osdmap e556: 8 total, 8 up, 8 in 2024-03-21T13:24:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:21.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:20 smithi089 ceph-mon[39209]: pgmap v857: 172 pgs: 172 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:21.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:20 smithi089 ceph-mon[39209]: osdmap e556: 8 total, 8 up, 8 in 2024-03-21T13:24:21.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:20 smithi089 ceph-mon[30095]: pgmap v857: 172 pgs: 172 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:20 smithi089 ceph-mon[30095]: osdmap e556: 8 total, 8 up, 8 in 2024-03-21T13:24:21.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:22 smithi152 ceph-mon[34051]: pgmap v860: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:22 smithi152 ceph-mon[34051]: osdmap e557: 8 total, 8 up, 8 in 2024-03-21T13:24:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:22 smithi152 ceph-mon[34051]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:22.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[30095]: pgmap v860: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:22.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:22.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[30095]: osdmap e557: 8 total, 8 up, 8 in 2024-03-21T13:24:22.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:22.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[30095]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:22.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[39209]: pgmap v860: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:22.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:22.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[39209]: osdmap e557: 8 total, 8 up, 8 in 2024-03-21T13:24:22.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:22.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:22 smithi089 ceph-mon[39209]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:23.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:23.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:23 smithi152 ceph-mon[34051]: osdmap e558: 8 total, 8 up, 8 in 2024-03-21T13:24:23.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-109", "overlaypool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:23.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-109", "overlaypool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:23.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:23 smithi152 ceph-mon[34051]: osdmap e559: 8 total, 8 up, 8 in 2024-03-21T13:24:23.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-109-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:24:23.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:23.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[39209]: osdmap e558: 8 total, 8 up, 8 in 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-109", "overlaypool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-109", "overlaypool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[39209]: osdmap e559: 8 total, 8 up, 8 in 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-109-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[30095]: osdmap e558: 8 total, 8 up, 8 in 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-109", "overlaypool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-109", "overlaypool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[30095]: osdmap e559: 8 total, 8 up, 8 in 2024-03-21T13:24:23.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-109-cache", "mode": "writeback"}]: dispatch 2024-03-21T13:24:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:24 smithi152 ceph-mon[34051]: pgmap v863: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:24 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-109-cache", "mode": "writeback"}]': finished 2024-03-21T13:24:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:24 smithi152 ceph-mon[34051]: osdmap e560: 8 total, 8 up, 8 in 2024-03-21T13:24:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:24:24.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[30095]: pgmap v863: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:24.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:24.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-109-cache", "mode": "writeback"}]': finished 2024-03-21T13:24:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[30095]: osdmap e560: 8 total, 8 up, 8 in 2024-03-21T13:24:24.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:24:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[39209]: pgmap v863: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-109-cache", "mode": "writeback"}]': finished 2024-03-21T13:24:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[39209]: osdmap e560: 8 total, 8 up, 8 in 2024-03-21T13:24:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:24:26.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:26 smithi152 ceph-mon[34051]: pgmap v866: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:26.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:24:26.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:26 smithi152 ceph-mon[34051]: osdmap e561: 8 total, 8 up, 8 in 2024-03-21T13:24:26.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:24:26.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[30095]: pgmap v866: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:26.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:24:26.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[30095]: osdmap e561: 8 total, 8 up, 8 in 2024-03-21T13:24:26.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:24:26.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[39209]: pgmap v866: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:26.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:24:26.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[39209]: osdmap e561: 8 total, 8 up, 8 in 2024-03-21T13:24:26.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:24:27.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:24:27.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:27 smithi152 ceph-mon[34051]: osdmap e562: 8 total, 8 up, 8 in 2024-03-21T13:24:27.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:24:27.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:27 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:27.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:24:27.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:27 smithi152 ceph-mon[34051]: osdmap e563: 8 total, 8 up, 8 in 2024-03-21T13:24:27.827 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:24:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:24:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:24:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[39209]: osdmap e562: 8 total, 8 up, 8 in 2024-03-21T13:24:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:24:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:24:27.827 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[39209]: osdmap e563: 8 total, 8 up, 8 in 2024-03-21T13:24:27.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:24:27.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[30095]: osdmap e562: 8 total, 8 up, 8 in 2024-03-21T13:24:27.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:24:27.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:27.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:24:27.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:27 smithi089 ceph-mon[30095]: osdmap e563: 8 total, 8 up, 8 in 2024-03-21T13:24:28.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:24:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:24:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:28 smithi152 ceph-mon[34051]: pgmap v869: 212 pgs: 212 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:28.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-21T13:24:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-21T13:24:28.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:28 smithi152 ceph-mon[34051]: osdmap e564: 8 total, 8 up, 8 in 2024-03-21T13:24:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[30095]: pgmap v869: 212 pgs: 212 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[30095]: osdmap e564: 8 total, 8 up, 8 in 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[39209]: pgmap v869: 212 pgs: 212 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-21T13:24:28.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:28 smithi089 ceph-mon[39209]: osdmap e564: 8 total, 8 up, 8 in 2024-03-21T13:24:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:29.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:29.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:29.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:29.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:30 smithi152 ceph-mon[34051]: pgmap v872: 212 pgs: 212 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:30 smithi152 ceph-mon[34051]: osdmap e565: 8 total, 8 up, 8 in 2024-03-21T13:24:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:30 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[30095]: pgmap v872: 212 pgs: 212 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[30095]: osdmap e565: 8 total, 8 up, 8 in 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[39209]: pgmap v872: 212 pgs: 212 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[39209]: osdmap e565: 8 total, 8 up, 8 in 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]: dispatch 2024-03-21T13:24:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:30 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:31.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:31 smithi152 ceph-mon[34051]: osdmap e566: 8 total, 8 up, 8 in 2024-03-21T13:24:31.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:31.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:31 smithi089 ceph-mon[30095]: osdmap e566: 8 total, 8 up, 8 in 2024-03-21T13:24:31.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-109", "tierpool": "test-rados-api-smithi089-79633-109-cache"}]': finished 2024-03-21T13:24:31.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:31 smithi089 ceph-mon[39209]: osdmap e566: 8 total, 8 up, 8 in 2024-03-21T13:24:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:32 smithi152 ceph-mon[34051]: pgmap v875: 212 pgs: 212 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:24:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:32 smithi152 ceph-mon[34051]: osdmap e567: 8 total, 8 up, 8 in 2024-03-21T13:24:32.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:32 smithi089 ceph-mon[39209]: pgmap v875: 212 pgs: 212 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:24:32.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:32 smithi089 ceph-mon[39209]: osdmap e567: 8 total, 8 up, 8 in 2024-03-21T13:24:32.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:32 smithi089 ceph-mon[30095]: pgmap v875: 212 pgs: 212 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:24:32.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:32 smithi089 ceph-mon[30095]: osdmap e567: 8 total, 8 up, 8 in 2024-03-21T13:24:33.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:33 smithi152 ceph-mon[34051]: osdmap e568: 8 total, 8 up, 8 in 2024-03-21T13:24:33.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:33.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:33.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:33 smithi152 ceph-mon[34051]: osdmap e569: 8 total, 8 up, 8 in 2024-03-21T13:24:33.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[30095]: osdmap e568: 8 total, 8 up, 8 in 2024-03-21T13:24:33.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:33.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:33.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[30095]: osdmap e569: 8 total, 8 up, 8 in 2024-03-21T13:24:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[39209]: osdmap e568: 8 total, 8 up, 8 in 2024-03-21T13:24:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:33.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:33 smithi089 ceph-mon[39209]: osdmap e569: 8 total, 8 up, 8 in 2024-03-21T13:24:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:34 smithi152 ceph-mon[34051]: pgmap v878: 172 pgs: 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:34 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:24:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:34.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:34 smithi152 ceph-mon[34051]: osdmap e570: 8 total, 8 up, 8 in 2024-03-21T13:24:34.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[30095]: pgmap v878: 172 pgs: 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:34.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:34.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:24:34.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:34.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[30095]: osdmap e570: 8 total, 8 up, 8 in 2024-03-21T13:24:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[39209]: pgmap v878: 172 pgs: 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]: dispatch 2024-03-21T13:24:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:24:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/45682582' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-109"}]': finished 2024-03-21T13:24:34.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:34 smithi089 ceph-mon[39209]: osdmap e570: 8 total, 8 up, 8 in 2024-03-21T13:24:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:35 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:35 smithi152 ceph-mon[34051]: osdmap e571: 8 total, 8 up, 8 in 2024-03-21T13:24:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[30095]: osdmap e571: 8 total, 8 up, 8 in 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[39209]: osdmap e571: 8 total, 8 up, 8 in 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:35.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:36.793 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:36 smithi152 ceph-mon[34051]: pgmap v881: 172 pgs: 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:36.794 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:36 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:36.794 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:36 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:36.794 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:36.794 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:36 smithi152 ceph-mon[34051]: osdmap e572: 8 total, 8 up, 8 in 2024-03-21T13:24:36.794 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:36.794 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:36.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[30095]: pgmap v881: 172 pgs: 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:36.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[30095]: osdmap e572: 8 total, 8 up, 8 in 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[39209]: pgmap v881: 172 pgs: 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[39209]: osdmap e572: 8 total, 8 up, 8 in 2024-03-21T13:24:36.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3817079121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:36.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]: dispatch 2024-03-21T13:24:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:24:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:24:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:24:38.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:24:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: pgmap v884: 164 pgs: 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:24:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: osdmap e573: 8 total, 8 up, 8 in 2024-03-21T13:24:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:38.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:38 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:38.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: pgmap v884: 164 pgs: 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: osdmap e573: 8 total, 8 up, 8 in 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:38.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: pgmap v884: 164 pgs: 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi089-79633-104"}]': finished 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: osdmap e573: 8 total, 8 up, 8 in 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:38.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:38 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:40.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:40.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:40.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: osdmap e574: 8 total, 8 up, 8 in 2024-03-21T13:24:40.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:40.073 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:40.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:24:40.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:24:40.074 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:39 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: osdmap e574: 8 total, 8 up, 8 in 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:24:40.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi089-79633-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: osdmap e574: 8 total, 8 up, 8 in 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:24:40.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:39 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:24:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:40 smithi152 ceph-mon[34051]: pgmap v887: 164 pgs: 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:24:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:40 smithi152 ceph-mon[34051]: osdmap e575: 8 total, 8 up, 8 in 2024-03-21T13:24:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:40 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:41.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:40 smithi152 ceph-mon[34051]: osdmap e576: 8 total, 8 up, 8 in 2024-03-21T13:24:41.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[30095]: pgmap v887: 164 pgs: 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:24:41.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[30095]: osdmap e575: 8 total, 8 up, 8 in 2024-03-21T13:24:41.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:41.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:41.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[30095]: osdmap e576: 8 total, 8 up, 8 in 2024-03-21T13:24:41.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[39209]: pgmap v887: 164 pgs: 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:24:41.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[39209]: osdmap e575: 8 total, 8 up, 8 in 2024-03-21T13:24:41.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:41.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi089-79633-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:41.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:40 smithi089 ceph-mon[39209]: osdmap e576: 8 total, 8 up, 8 in 2024-03-21T13:24:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:42 smithi152 ceph-mon[34051]: pgmap v890: 172 pgs: 7 creating+peering, 1 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:42 smithi152 ceph-mon[34051]: osdmap e577: 8 total, 8 up, 8 in 2024-03-21T13:24:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[30095]: pgmap v890: 172 pgs: 7 creating+peering, 1 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[30095]: osdmap e577: 8 total, 8 up, 8 in 2024-03-21T13:24:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[39209]: pgmap v890: 172 pgs: 7 creating+peering, 1 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[39209]: osdmap e577: 8 total, 8 up, 8 in 2024-03-21T13:24:42.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:42.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:43 smithi152 ceph-mon[34051]: osdmap e578: 8 total, 8 up, 8 in 2024-03-21T13:24:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:43.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[30095]: osdmap e578: 8 total, 8 up, 8 in 2024-03-21T13:24:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[39209]: osdmap e578: 8 total, 8 up, 8 in 2024-03-21T13:24:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:44 smithi152 ceph-mon[34051]: pgmap v893: 204 pgs: 7 creating+peering, 33 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:24:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:44 smithi152 ceph-mon[34051]: osdmap e579: 8 total, 8 up, 8 in 2024-03-21T13:24:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[30095]: pgmap v893: 204 pgs: 7 creating+peering, 33 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:24:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[30095]: osdmap e579: 8 total, 8 up, 8 in 2024-03-21T13:24:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:44.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[39209]: pgmap v893: 204 pgs: 7 creating+peering, 33 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:24:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[39209]: osdmap e579: 8 total, 8 up, 8 in 2024-03-21T13:24:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]': finished 2024-03-21T13:24:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:45 smithi152 ceph-mon[34051]: osdmap e580: 8 total, 8 up, 8 in 2024-03-21T13:24:45.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:45.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:45.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:45 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:45.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]': finished 2024-03-21T13:24:45.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[30095]: osdmap e580: 8 total, 8 up, 8 in 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-112"}]': finished 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[39209]: osdmap e580: 8 total, 8 up, 8 in 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:45 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:46 smithi152 ceph-mon[34051]: pgmap v896: 204 pgs: 7 creating+peering, 33 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:46 smithi152 ceph-mon[34051]: osdmap e581: 8 total, 8 up, 8 in 2024-03-21T13:24:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[30095]: pgmap v896: 204 pgs: 7 creating+peering, 33 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[30095]: osdmap e581: 8 total, 8 up, 8 in 2024-03-21T13:24:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[39209]: pgmap v896: 204 pgs: 7 creating+peering, 33 unknown, 164 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[39209]: osdmap e581: 8 total, 8 up, 8 in 2024-03-21T13:24:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:46.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]: dispatch 2024-03-21T13:24:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]': finished 2024-03-21T13:24:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:47 smithi152 ceph-mon[34051]: osdmap e582: 8 total, 8 up, 8 in 2024-03-21T13:24:47.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]': finished 2024-03-21T13:24:47.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:47 smithi089 ceph-mon[39209]: osdmap e582: 8 total, 8 up, 8 in 2024-03-21T13:24:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-112"}]': finished 2024-03-21T13:24:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:47 smithi089 ceph-mon[30095]: osdmap e582: 8 total, 8 up, 8 in 2024-03-21T13:24:47.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:24:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:24:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:24:48.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:24:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:24:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:48 smithi152 ceph-mon[34051]: pgmap v899: 204 pgs: 204 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:24:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:48 smithi152 ceph-mon[34051]: osdmap e583: 8 total, 8 up, 8 in 2024-03-21T13:24:48.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:48 smithi089 ceph-mon[39209]: pgmap v899: 204 pgs: 204 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:24:48.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:48 smithi089 ceph-mon[39209]: osdmap e583: 8 total, 8 up, 8 in 2024-03-21T13:24:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:48 smithi089 ceph-mon[30095]: pgmap v899: 204 pgs: 204 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:24:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:48 smithi089 ceph-mon[30095]: osdmap e583: 8 total, 8 up, 8 in 2024-03-21T13:24:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:49 smithi152 ceph-mon[34051]: osdmap e584: 8 total, 8 up, 8 in 2024-03-21T13:24:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[39209]: osdmap e584: 8 total, 8 up, 8 in 2024-03-21T13:24:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:49.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:24:49.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[30095]: osdmap e584: 8 total, 8 up, 8 in 2024-03-21T13:24:49.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:49.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:50 smithi152 ceph-mon[34051]: pgmap v902: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:24:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:50 smithi152 ceph-mon[34051]: osdmap e585: 8 total, 8 up, 8 in 2024-03-21T13:24:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:50 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:50.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[30095]: pgmap v902: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:24:50.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[30095]: osdmap e585: 8 total, 8 up, 8 in 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[39209]: pgmap v902: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[39209]: osdmap e585: 8 total, 8 up, 8 in 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:24:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:50 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:51.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:24:51.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[30095]: osdmap e586: 8 total, 8 up, 8 in 2024-03-21T13:24:51.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:51.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:51.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:24:51.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[39209]: osdmap e586: 8 total, 8 up, 8 in 2024-03-21T13:24:51.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:51.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:24:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:51 smithi152 ceph-mon[34051]: osdmap e586: 8 total, 8 up, 8 in 2024-03-21T13:24:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:52.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:52 smithi152 ceph-mon[34051]: pgmap v905: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:24:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]': finished 2024-03-21T13:24:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:52 smithi152 ceph-mon[34051]: osdmap e587: 8 total, 8 up, 8 in 2024-03-21T13:24:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]: dispatch 2024-03-21T13:24:53.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]: dispatch 2024-03-21T13:24:53.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[30095]: pgmap v905: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:24:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]': finished 2024-03-21T13:24:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[30095]: osdmap e587: 8 total, 8 up, 8 in 2024-03-21T13:24:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]: dispatch 2024-03-21T13:24:53.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]: dispatch 2024-03-21T13:24:53.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[39209]: pgmap v905: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:24:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-114"}]': finished 2024-03-21T13:24:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[39209]: osdmap e587: 8 total, 8 up, 8 in 2024-03-21T13:24:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]: dispatch 2024-03-21T13:24:53.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]: dispatch 2024-03-21T13:24:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:53 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]': finished 2024-03-21T13:24:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:53 smithi152 ceph-mon[34051]: osdmap e588: 8 total, 8 up, 8 in 2024-03-21T13:24:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:54.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]': finished 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[30095]: osdmap e588: 8 total, 8 up, 8 in 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-114", "mode": "writeback"}]': finished 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[39209]: osdmap e588: 8 total, 8 up, 8 in 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:24:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:54 smithi152 ceph-mon[34051]: pgmap v908: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:24:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:54 smithi152 ceph-mon[34051]: osdmap e589: 8 total, 8 up, 8 in 2024-03-21T13:24:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:55.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[30095]: pgmap v908: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:24:55.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[30095]: osdmap e589: 8 total, 8 up, 8 in 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[39209]: pgmap v908: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[39209]: osdmap e589: 8 total, 8 up, 8 in 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]: dispatch 2024-03-21T13:24:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:55 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]': finished 2024-03-21T13:24:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:55 smithi152 ceph-mon[34051]: osdmap e590: 8 total, 8 up, 8 in 2024-03-21T13:24:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:55 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]': finished 2024-03-21T13:24:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[30095]: osdmap e590: 8 total, 8 up, 8 in 2024-03-21T13:24:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:24:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-114"}]': finished 2024-03-21T13:24:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[39209]: osdmap e590: 8 total, 8 up, 8 in 2024-03-21T13:24:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:55 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:24:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:56 smithi152 ceph-mon[34051]: pgmap v911: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:56 smithi152 ceph-mon[34051]: osdmap e591: 8 total, 8 up, 8 in 2024-03-21T13:24:57.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:56 smithi089 ceph-mon[39209]: pgmap v911: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:56 smithi089 ceph-mon[39209]: osdmap e591: 8 total, 8 up, 8 in 2024-03-21T13:24:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:56 smithi089 ceph-mon[30095]: pgmap v911: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:24:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:56 smithi089 ceph-mon[30095]: osdmap e591: 8 total, 8 up, 8 in 2024-03-21T13:24:57.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:57 smithi089 ceph-mon[30095]: osdmap e592: 8 total, 8 up, 8 in 2024-03-21T13:24:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:24:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:24:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:24:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:57 smithi089 ceph-mon[39209]: osdmap e592: 8 total, 8 up, 8 in 2024-03-21T13:24:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:58.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:57 smithi152 ceph-mon[34051]: osdmap e592: 8 total, 8 up, 8 in 2024-03-21T13:24:58.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:58.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:24:58.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:24:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:24:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:58 smithi152 ceph-mon[34051]: pgmap v914: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:58 smithi152 ceph-mon[34051]: osdmap e593: 8 total, 8 up, 8 in 2024-03-21T13:24:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:59.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[30095]: pgmap v914: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:59.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:59.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[30095]: osdmap e593: 8 total, 8 up, 8 in 2024-03-21T13:24:59.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:24:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[39209]: pgmap v914: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:24:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:24:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[39209]: osdmap e593: 8 total, 8 up, 8 in 2024-03-21T13:24:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:59 smithi152 ceph-mon[34051]: osdmap e594: 8 total, 8 up, 8 in 2024-03-21T13:25:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:24:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:00.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:00.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[30095]: osdmap e594: 8 total, 8 up, 8 in 2024-03-21T13:25:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:00.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:00.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:00.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[39209]: osdmap e594: 8 total, 8 up, 8 in 2024-03-21T13:25:00.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:00.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:24:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:00 smithi152 ceph-mon[34051]: pgmap v917: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:00 smithi152 ceph-mon[34051]: osdmap e595: 8 total, 8 up, 8 in 2024-03-21T13:25:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:01.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[30095]: pgmap v917: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:01.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:01.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[30095]: osdmap e595: 8 total, 8 up, 8 in 2024-03-21T13:25:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:01.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[39209]: pgmap v917: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[39209]: osdmap e595: 8 total, 8 up, 8 in 2024-03-21T13:25:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]': finished 2024-03-21T13:25:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:01 smithi152 ceph-mon[34051]: osdmap e596: 8 total, 8 up, 8 in 2024-03-21T13:25:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]: dispatch 2024-03-21T13:25:02.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]: dispatch 2024-03-21T13:25:02.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]': finished 2024-03-21T13:25:02.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[30095]: osdmap e596: 8 total, 8 up, 8 in 2024-03-21T13:25:02.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]: dispatch 2024-03-21T13:25:02.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]: dispatch 2024-03-21T13:25:02.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-116"}]': finished 2024-03-21T13:25:02.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[39209]: osdmap e596: 8 total, 8 up, 8 in 2024-03-21T13:25:02.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]: dispatch 2024-03-21T13:25:02.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]: dispatch 2024-03-21T13:25:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:02 smithi152 ceph-mon[34051]: pgmap v920: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:02 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]': finished 2024-03-21T13:25:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:02 smithi152 ceph-mon[34051]: osdmap e597: 8 total, 8 up, 8 in 2024-03-21T13:25:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-03-21T13:25:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:02 smithi152 ceph-mon[34051]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-03-21T13:25:03.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[30095]: pgmap v920: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]': finished 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[30095]: osdmap e597: 8 total, 8 up, 8 in 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[30095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[39209]: pgmap v920: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-116", "mode": "writeback"}]': finished 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[39209]: osdmap e597: 8 total, 8 up, 8 in 2024-03-21T13:25:03.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-03-21T13:25:03.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:02 smithi089 ceph-mon[39209]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-03-21T13:25:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:03 smithi152 ceph-mon[34051]: 326.2 deep-scrub starts 2024-03-21T13:25:04.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:03 smithi152 ceph-mon[34051]: 326.2 deep-scrub ok 2024-03-21T13:25:04.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:03 smithi089 ceph-mon[30095]: 326.2 deep-scrub starts 2024-03-21T13:25:04.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:03 smithi089 ceph-mon[30095]: 326.2 deep-scrub ok 2024-03-21T13:25:04.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:03 smithi089 ceph-mon[39209]: 326.2 deep-scrub starts 2024-03-21T13:25:04.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:03 smithi089 ceph-mon[39209]: 326.2 deep-scrub ok 2024-03-21T13:25:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:04 smithi152 ceph-mon[34051]: pgmap v922: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-21T13:25:05.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:04 smithi089 ceph-mon[39209]: pgmap v922: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-21T13:25:05.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:04 smithi089 ceph-mon[30095]: pgmap v922: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-21T13:25:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:06 smithi152 ceph-mon[34051]: pgmap v923: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:25:07.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:06 smithi089 ceph-mon[30095]: pgmap v923: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:25:07.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:06 smithi089 ceph-mon[39209]: pgmap v923: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:25:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:25:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:25:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:25:08.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:25:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:25:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:08 smithi152 ceph-mon[34051]: pgmap v924: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 976 B/s wr, 2 op/s 2024-03-21T13:25:09.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:08 smithi089 ceph-mon[30095]: pgmap v924: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 976 B/s wr, 2 op/s 2024-03-21T13:25:09.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:08 smithi089 ceph-mon[39209]: pgmap v924: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 976 B/s wr, 2 op/s 2024-03-21T13:25:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:10.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:10.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:10.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:10 smithi089 ceph-mon[30095]: pgmap v925: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 860 B/s wr, 2 op/s 2024-03-21T13:25:10.976 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:10 smithi089 ceph-mon[39209]: pgmap v925: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 860 B/s wr, 2 op/s 2024-03-21T13:25:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:10 smithi152 ceph-mon[34051]: pgmap v925: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 860 B/s wr, 2 op/s 2024-03-21T13:25:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:12 smithi152 ceph-mon[34051]: pgmap v926: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:13.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:12 smithi089 ceph-mon[30095]: pgmap v926: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:13.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:12 smithi089 ceph-mon[39209]: pgmap v926: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:14.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:14.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:14.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:14.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:14 smithi152 ceph-mon[34051]: pgmap v927: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:25:15.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:14 smithi089 ceph-mon[30095]: pgmap v927: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:25:15.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:14 smithi089 ceph-mon[39209]: pgmap v927: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:25:17.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:16 smithi152 ceph-mon[34051]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T13:25:17.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:16 smithi089 ceph-mon[30095]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T13:25:17.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:16 smithi089 ceph-mon[39209]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T13:25:17.751 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:25:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:25:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:25:18.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:17 smithi152 ceph-mon[34051]: osdmap e598: 8 total, 8 up, 8 in 2024-03-21T13:25:18.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:18.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:18.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:17 smithi089 ceph-mon[30095]: osdmap e598: 8 total, 8 up, 8 in 2024-03-21T13:25:18.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:18.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:18.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:17 smithi089 ceph-mon[39209]: osdmap e598: 8 total, 8 up, 8 in 2024-03-21T13:25:18.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:18.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:18.762 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:25:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:25:19.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:18 smithi152 ceph-mon[34051]: pgmap v930: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:25:19.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:19.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:18 smithi152 ceph-mon[34051]: osdmap e599: 8 total, 8 up, 8 in 2024-03-21T13:25:19.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:19.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:19.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[30095]: pgmap v930: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[30095]: osdmap e599: 8 total, 8 up, 8 in 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[39209]: pgmap v930: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[39209]: osdmap e599: 8 total, 8 up, 8 in 2024-03-21T13:25:19.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:19.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]: dispatch 2024-03-21T13:25:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:19 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]': finished 2024-03-21T13:25:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:19 smithi152 ceph-mon[34051]: osdmap e600: 8 total, 8 up, 8 in 2024-03-21T13:25:20.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:20.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:20.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]': finished 2024-03-21T13:25:20.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[30095]: osdmap e600: 8 total, 8 up, 8 in 2024-03-21T13:25:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-116"}]': finished 2024-03-21T13:25:20.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:19 smithi089 ceph-mon[39209]: osdmap e600: 8 total, 8 up, 8 in 2024-03-21T13:25:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:20 smithi152 ceph-mon[34051]: pgmap v933: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:25:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:20 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:20 smithi152 ceph-mon[34051]: osdmap e601: 8 total, 8 up, 8 in 2024-03-21T13:25:21.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:20 smithi089 ceph-mon[30095]: pgmap v933: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:25:21.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:20 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:21.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:20 smithi089 ceph-mon[30095]: osdmap e601: 8 total, 8 up, 8 in 2024-03-21T13:25:21.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:20 smithi089 ceph-mon[39209]: pgmap v933: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:25:21.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:20 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:21.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:20 smithi089 ceph-mon[39209]: osdmap e601: 8 total, 8 up, 8 in 2024-03-21T13:25:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:21 smithi152 ceph-mon[34051]: osdmap e602: 8 total, 8 up, 8 in 2024-03-21T13:25:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:22.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:22.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:21 smithi089 ceph-mon[39209]: osdmap e602: 8 total, 8 up, 8 in 2024-03-21T13:25:22.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:22.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:22.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:21 smithi089 ceph-mon[30095]: osdmap e602: 8 total, 8 up, 8 in 2024-03-21T13:25:22.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:22.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:23.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:22 smithi152 ceph-mon[34051]: pgmap v936: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-21T13:25:23.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:23.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:22 smithi152 ceph-mon[34051]: osdmap e603: 8 total, 8 up, 8 in 2024-03-21T13:25:23.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:22 smithi089 ceph-mon[39209]: pgmap v936: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-21T13:25:23.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:23.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:22 smithi089 ceph-mon[39209]: osdmap e603: 8 total, 8 up, 8 in 2024-03-21T13:25:23.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:22 smithi089 ceph-mon[30095]: pgmap v936: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-21T13:25:23.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:23.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:22 smithi089 ceph-mon[30095]: osdmap e603: 8 total, 8 up, 8 in 2024-03-21T13:25:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:23 smithi152 ceph-mon[34051]: osdmap e604: 8 total, 8 up, 8 in 2024-03-21T13:25:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:24.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:24.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:23 smithi089 ceph-mon[39209]: osdmap e604: 8 total, 8 up, 8 in 2024-03-21T13:25:24.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:24.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:24.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:23 smithi089 ceph-mon[30095]: osdmap e604: 8 total, 8 up, 8 in 2024-03-21T13:25:24.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:24.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:24 smithi152 ceph-mon[34051]: pgmap v939: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-21T13:25:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:24 smithi152 ceph-mon[34051]: osdmap e605: 8 total, 8 up, 8 in 2024-03-21T13:25:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:25.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[39209]: pgmap v939: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-21T13:25:25.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:25.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[39209]: osdmap e605: 8 total, 8 up, 8 in 2024-03-21T13:25:25.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:25.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[30095]: pgmap v939: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-21T13:25:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[30095]: osdmap e605: 8 total, 8 up, 8 in 2024-03-21T13:25:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:25.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]': finished 2024-03-21T13:25:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: osdmap e606: 8 total, 8 up, 8 in 2024-03-21T13:25:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]: dispatch 2024-03-21T13:25:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]: dispatch 2024-03-21T13:25:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:26.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]': finished 2024-03-21T13:25:26.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:25 smithi152 ceph-mon[34051]: osdmap e607: 8 total, 8 up, 8 in 2024-03-21T13:25:26.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]': finished 2024-03-21T13:25:26.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: osdmap e606: 8 total, 8 up, 8 in 2024-03-21T13:25:26.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]: dispatch 2024-03-21T13:25:26.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]: dispatch 2024-03-21T13:25:26.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:26.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]': finished 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[39209]: osdmap e607: 8 total, 8 up, 8 in 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-118"}]': finished 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: osdmap e606: 8 total, 8 up, 8 in 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]: dispatch 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]: dispatch 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:26.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:26.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-118", "mode": "writeback"}]': finished 2024-03-21T13:25:26.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:25 smithi089 ceph-mon[30095]: osdmap e607: 8 total, 8 up, 8 in 2024-03-21T13:25:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:26 smithi152 ceph-mon[34051]: pgmap v942: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:26 smithi152 ceph-mon[34051]: osdmap e608: 8 total, 8 up, 8 in 2024-03-21T13:25:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:27.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[30095]: pgmap v942: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:27.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[30095]: osdmap e608: 8 total, 8 up, 8 in 2024-03-21T13:25:27.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:27.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[39209]: pgmap v942: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[39209]: osdmap e608: 8 total, 8 up, 8 in 2024-03-21T13:25:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:25:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:25:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:25:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:28 smithi152 ceph-mon[34051]: pgmap v945: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:28 smithi152 ceph-mon[34051]: osdmap e609: 8 total, 8 up, 8 in 2024-03-21T13:25:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[30095]: pgmap v945: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[30095]: osdmap e609: 8 total, 8 up, 8 in 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[39209]: pgmap v945: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[39209]: osdmap e609: 8 total, 8 up, 8 in 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]: dispatch 2024-03-21T13:25:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:28.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:25:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:25:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:29 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]': finished 2024-03-21T13:25:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:29 smithi152 ceph-mon[34051]: osdmap e610: 8 total, 8 up, 8 in 2024-03-21T13:25:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:29.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:29.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]': finished 2024-03-21T13:25:29.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[39209]: osdmap e610: 8 total, 8 up, 8 in 2024-03-21T13:25:29.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-118"}]': finished 2024-03-21T13:25:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[30095]: osdmap e610: 8 total, 8 up, 8 in 2024-03-21T13:25:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:30 smithi152 ceph-mon[34051]: pgmap v948: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:30 smithi152 ceph-mon[34051]: osdmap e611: 8 total, 8 up, 8 in 2024-03-21T13:25:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:30 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:30.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:30 smithi089 ceph-mon[30095]: pgmap v948: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:30.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:30 smithi089 ceph-mon[30095]: osdmap e611: 8 total, 8 up, 8 in 2024-03-21T13:25:30.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:30 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:30 smithi089 ceph-mon[39209]: pgmap v948: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:30 smithi089 ceph-mon[39209]: osdmap e611: 8 total, 8 up, 8 in 2024-03-21T13:25:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:30 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:31 smithi152 ceph-mon[34051]: osdmap e612: 8 total, 8 up, 8 in 2024-03-21T13:25:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:31.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:31 smithi089 ceph-mon[30095]: osdmap e612: 8 total, 8 up, 8 in 2024-03-21T13:25:31.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:31.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:31.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:31 smithi089 ceph-mon[39209]: osdmap e612: 8 total, 8 up, 8 in 2024-03-21T13:25:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[30095]: pgmap v951: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[30095]: osdmap e613: 8 total, 8 up, 8 in 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[39209]: pgmap v951: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[39209]: osdmap e613: 8 total, 8 up, 8 in 2024-03-21T13:25:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:32.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:32 smithi152 ceph-mon[34051]: pgmap v951: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:32 smithi152 ceph-mon[34051]: osdmap e613: 8 total, 8 up, 8 in 2024-03-21T13:25:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:33.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[30095]: osdmap e614: 8 total, 8 up, 8 in 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]': finished 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[30095]: osdmap e615: 8 total, 8 up, 8 in 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[39209]: osdmap e614: 8 total, 8 up, 8 in 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]': finished 2024-03-21T13:25:33.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:33 smithi089 ceph-mon[39209]: osdmap e615: 8 total, 8 up, 8 in 2024-03-21T13:25:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:33 smithi152 ceph-mon[34051]: osdmap e614: 8 total, 8 up, 8 in 2024-03-21T13:25:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-120"}]': finished 2024-03-21T13:25:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:33 smithi152 ceph-mon[34051]: osdmap e615: 8 total, 8 up, 8 in 2024-03-21T13:25:34.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:34 smithi152 ceph-mon[34051]: pgmap v954: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:34.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]: dispatch 2024-03-21T13:25:34.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]: dispatch 2024-03-21T13:25:34.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:34 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:34.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]': finished 2024-03-21T13:25:34.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[39209]: pgmap v954: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:34.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]: dispatch 2024-03-21T13:25:34.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]: dispatch 2024-03-21T13:25:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]': finished 2024-03-21T13:25:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[30095]: pgmap v954: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]: dispatch 2024-03-21T13:25:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]: dispatch 2024-03-21T13:25:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-120", "mode": "writeback"}]': finished 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: osdmap e616: 8 total, 8 up, 8 in 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: osdmap e617: 8 total, 8 up, 8 in 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:35.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: osdmap e616: 8 total, 8 up, 8 in 2024-03-21T13:25:35.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:35.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:35.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: osdmap e617: 8 total, 8 up, 8 in 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: osdmap e616: 8 total, 8 up, 8 in 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: osdmap e617: 8 total, 8 up, 8 in 2024-03-21T13:25:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:35.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]: dispatch 2024-03-21T13:25:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:36 smithi152 ceph-mon[34051]: pgmap v957: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:36 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]': finished 2024-03-21T13:25:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:36 smithi152 ceph-mon[34051]: osdmap e618: 8 total, 8 up, 8 in 2024-03-21T13:25:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[30095]: pgmap v957: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]': finished 2024-03-21T13:25:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[30095]: osdmap e618: 8 total, 8 up, 8 in 2024-03-21T13:25:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[39209]: pgmap v957: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-120"}]': finished 2024-03-21T13:25:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:36 smithi089 ceph-mon[39209]: osdmap e618: 8 total, 8 up, 8 in 2024-03-21T13:25:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:25:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:25:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:25:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:38 smithi152 ceph-mon[34051]: pgmap v960: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:25:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:38 smithi152 ceph-mon[34051]: osdmap e619: 8 total, 8 up, 8 in 2024-03-21T13:25:38.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:38 smithi089 ceph-mon[30095]: pgmap v960: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:25:38.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:38 smithi089 ceph-mon[30095]: osdmap e619: 8 total, 8 up, 8 in 2024-03-21T13:25:38.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:38 smithi089 ceph-mon[39209]: pgmap v960: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-21T13:25:38.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:38 smithi089 ceph-mon[39209]: osdmap e619: 8 total, 8 up, 8 in 2024-03-21T13:25:38.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:25:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:25:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:39 smithi152 ceph-mon[34051]: osdmap e620: 8 total, 8 up, 8 in 2024-03-21T13:25:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:39 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:25:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:39.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:39 smithi152 ceph-mon[34051]: osdmap e621: 8 total, 8 up, 8 in 2024-03-21T13:25:39.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[30095]: osdmap e620: 8 total, 8 up, 8 in 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[30095]: osdmap e621: 8 total, 8 up, 8 in 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[39209]: osdmap e620: 8 total, 8 up, 8 in 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:25:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:39.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:39 smithi089 ceph-mon[39209]: osdmap e621: 8 total, 8 up, 8 in 2024-03-21T13:25:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: pgmap v963: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:25:40.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:25:40.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:40.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:40.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:40.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: osdmap e622: 8 total, 8 up, 8 in 2024-03-21T13:25:40.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:40.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:40.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: pgmap v963: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:40.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: osdmap e622: 8 total, 8 up, 8 in 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:40.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: pgmap v963: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: osdmap e622: 8 total, 8 up, 8 in 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:40.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:42 smithi152 ceph-mon[34051]: pgmap v966: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:25:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]': finished 2024-03-21T13:25:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]: dispatch 2024-03-21T13:25:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:42 smithi152 ceph-mon[34051]: osdmap e623: 8 total, 8 up, 8 in 2024-03-21T13:25:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]: dispatch 2024-03-21T13:25:42.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[30095]: pgmap v966: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]': finished 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]: dispatch 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[30095]: osdmap e623: 8 total, 8 up, 8 in 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]: dispatch 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[39209]: pgmap v966: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-122"}]': finished 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]: dispatch 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[39209]: osdmap e623: 8 total, 8 up, 8 in 2024-03-21T13:25:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]: dispatch 2024-03-21T13:25:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:43 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]': finished 2024-03-21T13:25:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:43 smithi152 ceph-mon[34051]: osdmap e624: 8 total, 8 up, 8 in 2024-03-21T13:25:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:43.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]': finished 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[30095]: osdmap e624: 8 total, 8 up, 8 in 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-122", "mode": "writeback"}]': finished 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[39209]: osdmap e624: 8 total, 8 up, 8 in 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:44 smithi152 ceph-mon[34051]: pgmap v969: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:44 smithi152 ceph-mon[34051]: osdmap e625: 8 total, 8 up, 8 in 2024-03-21T13:25:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:44.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[30095]: pgmap v969: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[30095]: osdmap e625: 8 total, 8 up, 8 in 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[39209]: pgmap v969: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[39209]: osdmap e625: 8 total, 8 up, 8 in 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]: dispatch 2024-03-21T13:25:45.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:45 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:45.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:45 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]': finished 2024-03-21T13:25:45.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:45 smithi152 ceph-mon[34051]: osdmap e626: 8 total, 8 up, 8 in 2024-03-21T13:25:45.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:45 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:45.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:45.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]': finished 2024-03-21T13:25:45.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[30095]: osdmap e626: 8 total, 8 up, 8 in 2024-03-21T13:25:45.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:45.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:45.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-122"}]': finished 2024-03-21T13:25:45.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[39209]: osdmap e626: 8 total, 8 up, 8 in 2024-03-21T13:25:45.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:45 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:46 smithi152 ceph-mon[34051]: pgmap v972: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:46 smithi152 ceph-mon[34051]: osdmap e627: 8 total, 8 up, 8 in 2024-03-21T13:25:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:46 smithi089 ceph-mon[30095]: pgmap v972: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:46.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:46 smithi089 ceph-mon[30095]: osdmap e627: 8 total, 8 up, 8 in 2024-03-21T13:25:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:46 smithi089 ceph-mon[39209]: pgmap v972: 204 pgs: 204 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:46 smithi089 ceph-mon[39209]: osdmap e627: 8 total, 8 up, 8 in 2024-03-21T13:25:47.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:47 smithi089 ceph-mon[30095]: osdmap e628: 8 total, 8 up, 8 in 2024-03-21T13:25:47.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:47 smithi089 ceph-mon[39209]: osdmap e628: 8 total, 8 up, 8 in 2024-03-21T13:25:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:47 smithi152 ceph-mon[34051]: osdmap e628: 8 total, 8 up, 8 in 2024-03-21T13:25:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:25:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:25:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:25:48.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:48 smithi152 ceph-mon[34051]: pgmap v975: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:48.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:48.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:48 smithi152 ceph-mon[34051]: osdmap e629: 8 total, 8 up, 8 in 2024-03-21T13:25:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:48 smithi089 ceph-mon[30095]: pgmap v975: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:48 smithi089 ceph-mon[30095]: osdmap e629: 8 total, 8 up, 8 in 2024-03-21T13:25:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:48 smithi089 ceph-mon[39209]: pgmap v975: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:48 smithi089 ceph-mon[39209]: osdmap e629: 8 total, 8 up, 8 in 2024-03-21T13:25:48.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:25:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:25:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:49 smithi152 ceph-mon[34051]: osdmap e630: 8 total, 8 up, 8 in 2024-03-21T13:25:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[30095]: osdmap e630: 8 total, 8 up, 8 in 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[39209]: osdmap e630: 8 total, 8 up, 8 in 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:49.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:50 smithi152 ceph-mon[34051]: pgmap v978: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:50 smithi152 ceph-mon[34051]: osdmap e631: 8 total, 8 up, 8 in 2024-03-21T13:25:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[30095]: pgmap v978: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[30095]: osdmap e631: 8 total, 8 up, 8 in 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[39209]: pgmap v978: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[39209]: osdmap e631: 8 total, 8 up, 8 in 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]': finished 2024-03-21T13:25:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:51 smithi152 ceph-mon[34051]: osdmap e632: 8 total, 8 up, 8 in 2024-03-21T13:25:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]: dispatch 2024-03-21T13:25:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]: dispatch 2024-03-21T13:25:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:51 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]': finished 2024-03-21T13:25:51.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:51 smithi152 ceph-mon[34051]: osdmap e633: 8 total, 8 up, 8 in 2024-03-21T13:25:51.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]': finished 2024-03-21T13:25:51.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[30095]: osdmap e632: 8 total, 8 up, 8 in 2024-03-21T13:25:51.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]: dispatch 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]: dispatch 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]': finished 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[30095]: osdmap e633: 8 total, 8 up, 8 in 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-124"}]': finished 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[39209]: osdmap e632: 8 total, 8 up, 8 in 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]: dispatch 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]: dispatch 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:25:51.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-124", "mode": "writeback"}]': finished 2024-03-21T13:25:51.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:51 smithi089 ceph-mon[39209]: osdmap e633: 8 total, 8 up, 8 in 2024-03-21T13:25:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:52 smithi152 ceph-mon[34051]: pgmap v981: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:52.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:52 smithi089 ceph-mon[30095]: pgmap v981: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:52 smithi089 ceph-mon[39209]: pgmap v981: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:53 smithi152 ceph-mon[34051]: osdmap e634: 8 total, 8 up, 8 in 2024-03-21T13:25:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:53 smithi089 ceph-mon[30095]: osdmap e634: 8 total, 8 up, 8 in 2024-03-21T13:25:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:53 smithi089 ceph-mon[39209]: osdmap e634: 8 total, 8 up, 8 in 2024-03-21T13:25:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:25:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:54 smithi152 ceph-mon[34051]: pgmap v984: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:54 smithi152 ceph-mon[34051]: osdmap e635: 8 total, 8 up, 8 in 2024-03-21T13:25:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:54.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[39209]: pgmap v984: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:54.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[39209]: osdmap e635: 8 total, 8 up, 8 in 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[30095]: pgmap v984: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[30095]: osdmap e635: 8 total, 8 up, 8 in 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]: dispatch 2024-03-21T13:25:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:55 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]': finished 2024-03-21T13:25:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:55 smithi152 ceph-mon[34051]: osdmap e636: 8 total, 8 up, 8 in 2024-03-21T13:25:55.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:55 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:55.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]': finished 2024-03-21T13:25:55.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:55 smithi089 ceph-mon[30095]: osdmap e636: 8 total, 8 up, 8 in 2024-03-21T13:25:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:55 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:25:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-124"}]': finished 2024-03-21T13:25:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:55 smithi089 ceph-mon[39209]: osdmap e636: 8 total, 8 up, 8 in 2024-03-21T13:25:56.472 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:56 smithi152 ceph-mon[34051]: pgmap v987: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:56.472 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:56 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:56 smithi152 ceph-mon[34051]: osdmap e637: 8 total, 8 up, 8 in 2024-03-21T13:25:56.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:56 smithi089 ceph-mon[30095]: pgmap v987: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:56.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:56 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:56 smithi089 ceph-mon[30095]: osdmap e637: 8 total, 8 up, 8 in 2024-03-21T13:25:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:56 smithi089 ceph-mon[39209]: pgmap v987: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:25:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:56 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:25:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:56 smithi089 ceph-mon[39209]: osdmap e637: 8 total, 8 up, 8 in 2024-03-21T13:25:57.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:57 smithi089 ceph-mon[30095]: osdmap e638: 8 total, 8 up, 8 in 2024-03-21T13:25:57.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:57.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:57 smithi089 ceph-mon[39209]: osdmap e638: 8 total, 8 up, 8 in 2024-03-21T13:25:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:57 smithi152 ceph-mon[34051]: osdmap e638: 8 total, 8 up, 8 in 2024-03-21T13:25:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:25:57.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:25:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:25:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:25:58.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:25:58.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-mon[34051]: pgmap v990: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-mon[34051]: osdmap e639: 8 total, 8 up, 8 in 2024-03-21T13:25:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:58.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:58.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[39209]: pgmap v990: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:58.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:58.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[39209]: osdmap e639: 8 total, 8 up, 8 in 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[30095]: pgmap v990: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[30095]: osdmap e639: 8 total, 8 up, 8 in 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:25:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:25:58.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:25:59.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:59.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:59 smithi152 ceph-mon[34051]: osdmap e640: 8 total, 8 up, 8 in 2024-03-21T13:25:59.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:25:59.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:25:59.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:25:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:59.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[30095]: osdmap e640: 8 total, 8 up, 8 in 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[39209]: osdmap e640: 8 total, 8 up, 8 in 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:25:59.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:25:59.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:25:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: pgmap v993: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]': finished 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: osdmap e641: 8 total, 8 up, 8 in 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]: dispatch 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]: dispatch 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]': finished 2024-03-21T13:26:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: osdmap e642: 8 total, 8 up, 8 in 2024-03-21T13:26:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:00.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:00.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: pgmap v993: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]': finished 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: osdmap e641: 8 total, 8 up, 8 in 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]: dispatch 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]: dispatch 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]': finished 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: osdmap e642: 8 total, 8 up, 8 in 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: pgmap v993: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:00.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-126"}]': finished 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: osdmap e641: 8 total, 8 up, 8 in 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]: dispatch 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]: dispatch 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-126", "mode": "writeback"}]': finished 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: osdmap e642: 8 total, 8 up, 8 in 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:00.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:02 smithi152 ceph-mon[34051]: pgmap v996: 204 pgs: 204 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:26:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:02 smithi152 ceph-mon[34051]: osdmap e643: 8 total, 8 up, 8 in 2024-03-21T13:26:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:26:02.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:26:02.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[30095]: pgmap v996: 204 pgs: 204 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:26:02.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[30095]: osdmap e643: 8 total, 8 up, 8 in 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[39209]: pgmap v996: 204 pgs: 204 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[39209]: osdmap e643: 8 total, 8 up, 8 in 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:26:02.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]: dispatch 2024-03-21T13:26:03.058 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9072 ms) 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6045 ms) 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5025 ms) 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (24479 ms) 2024-03-21T13:26:03.059 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13286 ms) 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12109 ms) 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: 00000012 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12055 ms) 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: require_osd_release = squid 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6047 ms) 2024-03-21T13:26:03.060 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (553870 ms total) 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1033 ms) 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11025 ms) 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18083 ms) 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30143 ms total) 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: 2024-03-21T13:26:03.061 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7048 ms) 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8056 ms) 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: waiting for scrub... 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: done waiting 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24164 ms) 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9277 ms) 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7981 ms) 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-21T13:26:03.062 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8049 ms) 2024-03-21T13:26:03.063 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-21T13:26:03.063 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10071 ms) 2024-03-21T13:26:03.063 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-21T13:26:03.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:03 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:03.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]': finished 2024-03-21T13:26:03.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:03 smithi152 ceph-mon[34051]: osdmap e644: 8 total, 8 up, 8 in 2024-03-21T13:26:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:03 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:03.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]': finished 2024-03-21T13:26:03.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:03 smithi089 ceph-mon[30095]: osdmap e644: 8 total, 8 up, 8 in 2024-03-21T13:26:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:03 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-126"}]': finished 2024-03-21T13:26:03.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:03 smithi089 ceph-mon[39209]: osdmap e644: 8 total, 8 up, 8 in 2024-03-21T13:26:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:04 smithi152 ceph-mon[34051]: pgmap v999: 204 pgs: 204 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:26:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:04 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:04 smithi152 ceph-mon[34051]: osdmap e645: 8 total, 8 up, 8 in 2024-03-21T13:26:04.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:04 smithi089 ceph-mon[30095]: pgmap v999: 204 pgs: 204 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:26:04.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:04 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:04.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:04 smithi089 ceph-mon[30095]: osdmap e645: 8 total, 8 up, 8 in 2024-03-21T13:26:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:04 smithi089 ceph-mon[39209]: pgmap v999: 204 pgs: 204 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T13:26:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:04 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:04 smithi089 ceph-mon[39209]: osdmap e645: 8 total, 8 up, 8 in 2024-03-21T13:26:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:05 smithi152 ceph-mon[34051]: osdmap e646: 8 total, 8 up, 8 in 2024-03-21T13:26:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:05.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:05 smithi089 ceph-mon[30095]: osdmap e646: 8 total, 8 up, 8 in 2024-03-21T13:26:05.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:05 smithi089 ceph-mon[39209]: osdmap e646: 8 total, 8 up, 8 in 2024-03-21T13:26:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[39209]: pgmap v1002: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[39209]: osdmap e647: 8 total, 8 up, 8 in 2024-03-21T13:26:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[30095]: pgmap v1002: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:06.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[30095]: osdmap e647: 8 total, 8 up, 8 in 2024-03-21T13:26:06.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:06.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:06 smithi152 ceph-mon[34051]: pgmap v1002: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:06 smithi152 ceph-mon[34051]: osdmap e647: 8 total, 8 up, 8 in 2024-03-21T13:26:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:07.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:07.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[30095]: osdmap e648: 8 total, 8 up, 8 in 2024-03-21T13:26:07.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:07.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:07.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:07.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[39209]: osdmap e648: 8 total, 8 up, 8 in 2024-03-21T13:26:07.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:07.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:07 smithi152 ceph-mon[34051]: osdmap e648: 8 total, 8 up, 8 in 2024-03-21T13:26:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:07 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:07.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:26:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:26:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:26:08.400 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:08 smithi152 ceph-mon[34051]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:26:08.400 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]': finished 2024-03-21T13:26:08.400 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:08 smithi152 ceph-mon[34051]: osdmap e649: 8 total, 8 up, 8 in 2024-03-21T13:26:08.400 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:08 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]: dispatch 2024-03-21T13:26:08.400 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:08 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]: dispatch 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[39209]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]': finished 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[39209]: osdmap e649: 8 total, 8 up, 8 in 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]: dispatch 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]: dispatch 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[30095]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-128"}]': finished 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[30095]: osdmap e649: 8 total, 8 up, 8 in 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]: dispatch 2024-03-21T13:26:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:08 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]: dispatch 2024-03-21T13:26:08.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:26:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:26:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:09 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:09 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]': finished 2024-03-21T13:26:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:09 smithi152 ceph-mon[34051]: osdmap e650: 8 total, 8 up, 8 in 2024-03-21T13:26:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:09.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:09.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]': finished 2024-03-21T13:26:09.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[30095]: osdmap e650: 8 total, 8 up, 8 in 2024-03-21T13:26:09.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:09.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:09.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-128", "mode": "writeback"}]': finished 2024-03-21T13:26:09.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[39209]: osdmap e650: 8 total, 8 up, 8 in 2024-03-21T13:26:09.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:10 smithi089 ceph-mon[39209]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:10 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:10.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:10 smithi089 ceph-mon[30095]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:10.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:10 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:10 smithi152 ceph-mon[34051]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:10 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:12 smithi152 ceph-mon[34051]: pgmap v1009: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 345 B/s wr, 2 op/s 2024-03-21T13:26:12.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:12 smithi089 ceph-mon[30095]: pgmap v1009: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 345 B/s wr, 2 op/s 2024-03-21T13:26:12.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:12 smithi089 ceph-mon[39209]: pgmap v1009: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 345 B/s wr, 2 op/s 2024-03-21T13:26:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:13 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:14.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:14.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:14.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:14.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:14.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:14.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:14.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:14.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:13 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:14 smithi152 ceph-mon[34051]: pgmap v1010: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 295 B/s wr, 1 op/s 2024-03-21T13:26:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:14 smithi152 ceph-mon[34051]: osdmap e651: 8 total, 8 up, 8 in 2024-03-21T13:26:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:15.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[30095]: pgmap v1010: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 295 B/s wr, 1 op/s 2024-03-21T13:26:15.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:15.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[30095]: osdmap e651: 8 total, 8 up, 8 in 2024-03-21T13:26:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:15.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:15.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[39209]: pgmap v1010: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 295 B/s wr, 1 op/s 2024-03-21T13:26:15.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:15.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[39209]: osdmap e651: 8 total, 8 up, 8 in 2024-03-21T13:26:15.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:15.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]: dispatch 2024-03-21T13:26:16.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:15 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:16.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]': finished 2024-03-21T13:26:16.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:15 smithi152 ceph-mon[34051]: osdmap e652: 8 total, 8 up, 8 in 2024-03-21T13:26:16.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:15 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:16.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]': finished 2024-03-21T13:26:16.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:15 smithi089 ceph-mon[30095]: osdmap e652: 8 total, 8 up, 8 in 2024-03-21T13:26:16.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:15 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:16.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-128"}]': finished 2024-03-21T13:26:16.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:15 smithi089 ceph-mon[39209]: osdmap e652: 8 total, 8 up, 8 in 2024-03-21T13:26:17.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:16 smithi152 ceph-mon[34051]: pgmap v1013: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 296 B/s wr, 1 op/s 2024-03-21T13:26:17.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:16 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:17.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:16 smithi152 ceph-mon[34051]: osdmap e653: 8 total, 8 up, 8 in 2024-03-21T13:26:17.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:16 smithi089 ceph-mon[30095]: pgmap v1013: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 296 B/s wr, 1 op/s 2024-03-21T13:26:17.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:16 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:17.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:16 smithi089 ceph-mon[30095]: osdmap e653: 8 total, 8 up, 8 in 2024-03-21T13:26:17.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:16 smithi089 ceph-mon[39209]: pgmap v1013: 204 pgs: 204 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 296 B/s wr, 1 op/s 2024-03-21T13:26:17.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:16 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:17.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:16 smithi089 ceph-mon[39209]: osdmap e653: 8 total, 8 up, 8 in 2024-03-21T13:26:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:26:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:26:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:26:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:17 smithi152 ceph-mon[34051]: osdmap e654: 8 total, 8 up, 8 in 2024-03-21T13:26:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:17 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:17 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:18.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:17 smithi089 ceph-mon[30095]: osdmap e654: 8 total, 8 up, 8 in 2024-03-21T13:26:18.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:17 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:18.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:17 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:17 smithi089 ceph-mon[39209]: osdmap e654: 8 total, 8 up, 8 in 2024-03-21T13:26:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:17 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:17 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:18.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:26:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:26:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:18 smithi152 ceph-mon[34051]: pgmap v1016: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:18 smithi152 ceph-mon[34051]: osdmap e655: 8 total, 8 up, 8 in 2024-03-21T13:26:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:19.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[30095]: pgmap v1016: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:19.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:19.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[30095]: osdmap e655: 8 total, 8 up, 8 in 2024-03-21T13:26:19.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[39209]: pgmap v1016: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[39209]: osdmap e655: 8 total, 8 up, 8 in 2024-03-21T13:26:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:19 smithi152 ceph-mon[34051]: osdmap e656: 8 total, 8 up, 8 in 2024-03-21T13:26:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:20.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:20.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:20.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[30095]: osdmap e656: 8 total, 8 up, 8 in 2024-03-21T13:26:20.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[39209]: osdmap e656: 8 total, 8 up, 8 in 2024-03-21T13:26:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:20 smithi152 ceph-mon[34051]: pgmap v1019: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]': finished 2024-03-21T13:26:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:20 smithi152 ceph-mon[34051]: osdmap e657: 8 total, 8 up, 8 in 2024-03-21T13:26:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]: dispatch 2024-03-21T13:26:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]: dispatch 2024-03-21T13:26:21.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[39209]: pgmap v1019: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:21.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]': finished 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[39209]: osdmap e657: 8 total, 8 up, 8 in 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]: dispatch 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]: dispatch 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[30095]: pgmap v1019: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-130"}]': finished 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[30095]: osdmap e657: 8 total, 8 up, 8 in 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]: dispatch 2024-03-21T13:26:21.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]: dispatch 2024-03-21T13:26:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:21 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]': finished 2024-03-21T13:26:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:21 smithi152 ceph-mon[34051]: osdmap e658: 8 total, 8 up, 8 in 2024-03-21T13:26:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:21 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:21 smithi152 ceph-mon[34051]: pgmap v1022: 204 pgs: 204 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:26:22.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:22.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]': finished 2024-03-21T13:26:22.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[39209]: osdmap e658: 8 total, 8 up, 8 in 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[39209]: pgmap v1022: 204 pgs: 204 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-130", "mode": "writeback"}]': finished 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[30095]: osdmap e658: 8 total, 8 up, 8 in 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:21 smithi089 ceph-mon[30095]: pgmap v1022: 204 pgs: 204 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:26:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:22 smithi152 ceph-mon[34051]: osdmap e659: 8 total, 8 up, 8 in 2024-03-21T13:26:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[30095]: osdmap e659: 8 total, 8 up, 8 in 2024-03-21T13:26:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[39209]: osdmap e659: 8 total, 8 up, 8 in 2024-03-21T13:26:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]: dispatch 2024-03-21T13:26:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:23 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]': finished 2024-03-21T13:26:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:23 smithi152 ceph-mon[34051]: osdmap e660: 8 total, 8 up, 8 in 2024-03-21T13:26:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:23 smithi152 ceph-mon[34051]: pgmap v1025: 204 pgs: 204 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:26:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]': finished 2024-03-21T13:26:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[30095]: osdmap e660: 8 total, 8 up, 8 in 2024-03-21T13:26:24.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[30095]: pgmap v1025: 204 pgs: 204 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:26:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-130"}]': finished 2024-03-21T13:26:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[39209]: osdmap e660: 8 total, 8 up, 8 in 2024-03-21T13:26:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:23 smithi089 ceph-mon[39209]: pgmap v1025: 204 pgs: 204 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-21T13:26:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:24 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:24 smithi152 ceph-mon[34051]: osdmap e661: 8 total, 8 up, 8 in 2024-03-21T13:26:25.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:24 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:25.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:24 smithi089 ceph-mon[39209]: osdmap e661: 8 total, 8 up, 8 in 2024-03-21T13:26:25.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:24 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:25.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:24 smithi089 ceph-mon[30095]: osdmap e661: 8 total, 8 up, 8 in 2024-03-21T13:26:26.222 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:25 smithi152 ceph-mon[34051]: osdmap e662: 8 total, 8 up, 8 in 2024-03-21T13:26:26.222 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:26.222 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:26.222 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:25 smithi152 ceph-mon[34051]: pgmap v1028: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:26.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[30095]: osdmap e662: 8 total, 8 up, 8 in 2024-03-21T13:26:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[30095]: pgmap v1028: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:26.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[39209]: osdmap e662: 8 total, 8 up, 8 in 2024-03-21T13:26:26.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:26.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:26.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:25 smithi089 ceph-mon[39209]: pgmap v1028: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:26 smithi152 ceph-mon[34051]: osdmap e663: 8 total, 8 up, 8 in 2024-03-21T13:26:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:27.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:27.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[30095]: osdmap e663: 8 total, 8 up, 8 in 2024-03-21T13:26:27.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:27.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[39209]: osdmap e663: 8 total, 8 up, 8 in 2024-03-21T13:26:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:26:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:26:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:27 smithi152 ceph-mon[34051]: osdmap e664: 8 total, 8 up, 8 in 2024-03-21T13:26:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:27 smithi152 ceph-mon[34051]: pgmap v1031: 204 pgs: 1 creating+activating, 13 creating+peering, 190 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:28.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:27 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:28.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:27 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[30095]: osdmap e664: 8 total, 8 up, 8 in 2024-03-21T13:26:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[30095]: pgmap v1031: 204 pgs: 1 creating+activating, 13 creating+peering, 190 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[39209]: osdmap e664: 8 total, 8 up, 8 in 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[39209]: pgmap v1031: 204 pgs: 1 creating+activating, 13 creating+peering, 190 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:27 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:28.822 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:26:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:26:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:28 smithi152 ceph-mon[34051]: osdmap e665: 8 total, 8 up, 8 in 2024-03-21T13:26:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]: dispatch 2024-03-21T13:26:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]: dispatch 2024-03-21T13:26:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:28 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:29.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:29.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[30095]: osdmap e665: 8 total, 8 up, 8 in 2024-03-21T13:26:29.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]: dispatch 2024-03-21T13:26:29.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]: dispatch 2024-03-21T13:26:29.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[39209]: osdmap e665: 8 total, 8 up, 8 in 2024-03-21T13:26:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]: dispatch 2024-03-21T13:26:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]: dispatch 2024-03-21T13:26:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:28 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:29 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]': finished 2024-03-21T13:26:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:29 smithi152 ceph-mon[34051]: osdmap e666: 8 total, 8 up, 8 in 2024-03-21T13:26:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:29 smithi152 ceph-mon[34051]: pgmap v1034: 204 pgs: 1 creating+activating, 13 creating+peering, 190 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]': finished 2024-03-21T13:26:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[39209]: osdmap e666: 8 total, 8 up, 8 in 2024-03-21T13:26:30.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[39209]: pgmap v1034: 204 pgs: 1 creating+activating, 13 creating+peering, 190 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:30.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:30.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-132", "mode": "writeback"}]': finished 2024-03-21T13:26:30.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[30095]: osdmap e666: 8 total, 8 up, 8 in 2024-03-21T13:26:30.326 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:29 smithi089 ceph-mon[30095]: pgmap v1034: 204 pgs: 1 creating+activating, 13 creating+peering, 190 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:30 smithi152 ceph-mon[34051]: osdmap e667: 8 total, 8 up, 8 in 2024-03-21T13:26:31.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:30 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:31.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:30 smithi089 ceph-mon[30095]: osdmap e667: 8 total, 8 up, 8 in 2024-03-21T13:26:31.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:30 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:30 smithi089 ceph-mon[39209]: osdmap e667: 8 total, 8 up, 8 in 2024-03-21T13:26:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:30 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:26:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:31 smithi152 ceph-mon[34051]: osdmap e668: 8 total, 8 up, 8 in 2024-03-21T13:26:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:31 smithi152 ceph-mon[34051]: pgmap v1037: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:26:32.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[30095]: osdmap e668: 8 total, 8 up, 8 in 2024-03-21T13:26:32.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:32.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[30095]: pgmap v1037: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:26:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[39209]: osdmap e668: 8 total, 8 up, 8 in 2024-03-21T13:26:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:32.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:31 smithi089 ceph-mon[39209]: pgmap v1037: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:26:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:32 smithi152 ceph-mon[34051]: osdmap e669: 8 total, 8 up, 8 in 2024-03-21T13:26:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:33.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:33.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[30095]: osdmap e669: 8 total, 8 up, 8 in 2024-03-21T13:26:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:33.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[39209]: osdmap e669: 8 total, 8 up, 8 in 2024-03-21T13:26:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:33.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:33 smithi152 ceph-mon[34051]: osdmap e670: 8 total, 8 up, 8 in 2024-03-21T13:26:34.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:33 smithi152 ceph-mon[34051]: pgmap v1040: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:26:34.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:34.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:33 smithi089 ceph-mon[30095]: osdmap e670: 8 total, 8 up, 8 in 2024-03-21T13:26:34.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:33 smithi089 ceph-mon[30095]: pgmap v1040: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:26:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:33 smithi089 ceph-mon[39209]: osdmap e670: 8 total, 8 up, 8 in 2024-03-21T13:26:34.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:33 smithi089 ceph-mon[39209]: pgmap v1040: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-21T13:26:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:34 smithi152 ceph-mon[34051]: osdmap e671: 8 total, 8 up, 8 in 2024-03-21T13:26:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:35.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:35.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:34 smithi089 ceph-mon[30095]: osdmap e671: 8 total, 8 up, 8 in 2024-03-21T13:26:35.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:35.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:35.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:34 smithi089 ceph-mon[39209]: osdmap e671: 8 total, 8 up, 8 in 2024-03-21T13:26:35.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:35.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:36.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:36.335 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:35 smithi152 ceph-mon[34051]: osdmap e672: 8 total, 8 up, 8 in 2024-03-21T13:26:36.335 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:36.335 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:36.335 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:35 smithi152 ceph-mon[34051]: pgmap v1043: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:36.336 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:36.336 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[39209]: osdmap e672: 8 total, 8 up, 8 in 2024-03-21T13:26:36.336 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:36.336 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:36.336 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[39209]: pgmap v1043: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:36.336 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:36.336 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[30095]: osdmap e672: 8 total, 8 up, 8 in 2024-03-21T13:26:36.337 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:36.337 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]: dispatch 2024-03-21T13:26:36.337 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:35 smithi089 ceph-mon[30095]: pgmap v1043: 204 pgs: 204 active+clean; 582 KiB data, 686 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:36 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:37.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:36 smithi152 ceph-mon[34051]: osdmap e673: 8 total, 8 up, 8 in 2024-03-21T13:26:37.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:36 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:37.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:37.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:36 smithi089 ceph-mon[39209]: osdmap e673: 8 total, 8 up, 8 in 2024-03-21T13:26:37.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:36 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-132"}]': finished 2024-03-21T13:26:37.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:36 smithi089 ceph-mon[30095]: osdmap e673: 8 total, 8 up, 8 in 2024-03-21T13:26:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:26:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:26:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:26:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:37 smithi152 ceph-mon[34051]: osdmap e674: 8 total, 8 up, 8 in 2024-03-21T13:26:38.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:37 smithi152 ceph-mon[34051]: pgmap v1046: 172 pgs: 172 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-21T13:26:38.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:37 smithi089 ceph-mon[39209]: osdmap e674: 8 total, 8 up, 8 in 2024-03-21T13:26:38.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:37 smithi089 ceph-mon[39209]: pgmap v1046: 172 pgs: 172 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-21T13:26:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:37 smithi089 ceph-mon[30095]: osdmap e674: 8 total, 8 up, 8 in 2024-03-21T13:26:38.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:37 smithi089 ceph-mon[30095]: pgmap v1046: 172 pgs: 172 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-21T13:26:38.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:26:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:26:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:38 smithi152 ceph-mon[34051]: osdmap e675: 8 total, 8 up, 8 in 2024-03-21T13:26:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:39.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:38 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:39.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[30095]: osdmap e675: 8 total, 8 up, 8 in 2024-03-21T13:26:39.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:39.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:39.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[39209]: osdmap e675: 8 total, 8 up, 8 in 2024-03-21T13:26:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:39.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:38 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:40.248 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[39209]: osdmap e676: 8 total, 8 up, 8 in 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[39209]: pgmap v1049: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[30095]: osdmap e676: 8 total, 8 up, 8 in 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[30095]: pgmap v1049: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-21T13:26:40.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:39 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:26:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:39 smithi152 ceph-mon[34051]: osdmap e676: 8 total, 8 up, 8 in 2024-03-21T13:26:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:39 smithi152 ceph-mon[34051]: pgmap v1049: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-21T13:26:40.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:39 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: osdmap e677: 8 total, 8 up, 8 in 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]': finished 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: osdmap e678: 8 total, 8 up, 8 in 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]: dispatch 2024-03-21T13:26:41.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:26:41.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]: dispatch 2024-03-21T13:26:41.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:26:41.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:40 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:41.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:41.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: osdmap e677: 8 total, 8 up, 8 in 2024-03-21T13:26:41.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]': finished 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: osdmap e678: 8 total, 8 up, 8 in 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]: dispatch 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]: dispatch 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: osdmap e677: 8 total, 8 up, 8 in 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:41.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-134"}]': finished 2024-03-21T13:26:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: osdmap e678: 8 total, 8 up, 8 in 2024-03-21T13:26:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]: dispatch 2024-03-21T13:26:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:26:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]: dispatch 2024-03-21T13:26:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:26:41.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:40 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:42 smithi152 ceph-mon[34051]: pgmap v1052: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:42 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]': finished 2024-03-21T13:26:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:42 smithi152 ceph-mon[34051]: osdmap e679: 8 total, 8 up, 8 in 2024-03-21T13:26:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:42 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:42.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:42.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[30095]: pgmap v1052: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:42.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:42.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]': finished 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[30095]: osdmap e679: 8 total, 8 up, 8 in 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[39209]: pgmap v1052: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-134", "mode": "writeback"}]': finished 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[39209]: osdmap e679: 8 total, 8 up, 8 in 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:42.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:43 smithi152 ceph-mon[34051]: osdmap e680: 8 total, 8 up, 8 in 2024-03-21T13:26:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:43 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:43 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:43.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:43 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:43.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:43.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:43.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[30095]: osdmap e680: 8 total, 8 up, 8 in 2024-03-21T13:26:43.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[39209]: osdmap e680: 8 total, 8 up, 8 in 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]: dispatch 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:43.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:44 smithi152 ceph-mon[34051]: pgmap v1055: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:44 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:44 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]': finished 2024-03-21T13:26:44.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:44 smithi152 ceph-mon[34051]: osdmap e681: 8 total, 8 up, 8 in 2024-03-21T13:26:44.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[30095]: pgmap v1055: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:44.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:44.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]': finished 2024-03-21T13:26:44.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[30095]: osdmap e681: 8 total, 8 up, 8 in 2024-03-21T13:26:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[39209]: pgmap v1055: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-134"}]': finished 2024-03-21T13:26:44.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:44 smithi089 ceph-mon[39209]: osdmap e681: 8 total, 8 up, 8 in 2024-03-21T13:26:45.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:45 smithi089 ceph-mon[30095]: osdmap e682: 8 total, 8 up, 8 in 2024-03-21T13:26:45.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:45 smithi089 ceph-mon[39209]: osdmap e682: 8 total, 8 up, 8 in 2024-03-21T13:26:45.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:45 smithi152 ceph-mon[34051]: osdmap e682: 8 total, 8 up, 8 in 2024-03-21T13:26:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:46 smithi152 ceph-mon[34051]: pgmap v1058: 172 pgs: 172 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:46.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:46 smithi152 ceph-mon[34051]: osdmap e683: 8 total, 8 up, 8 in 2024-03-21T13:26:46.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:46.576 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:46.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[30095]: pgmap v1058: 172 pgs: 172 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:46.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[30095]: osdmap e683: 8 total, 8 up, 8 in 2024-03-21T13:26:46.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:46.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:46.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[39209]: pgmap v1058: 172 pgs: 172 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:46.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[39209]: osdmap e683: 8 total, 8 up, 8 in 2024-03-21T13:26:46.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:46.577 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[39209]: osdmap e684: 8 total, 8 up, 8 in 2024-03-21T13:26:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:47.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[39209]: osdmap e685: 8 total, 8 up, 8 in 2024-03-21T13:26:47.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:47.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[30095]: osdmap e684: 8 total, 8 up, 8 in 2024-03-21T13:26:47.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:47.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:47.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:47.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-mon[30095]: osdmap e685: 8 total, 8 up, 8 in 2024-03-21T13:26:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:47 smithi152 ceph-mon[34051]: osdmap e684: 8 total, 8 up, 8 in 2024-03-21T13:26:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:47.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:47 smithi152 ceph-mon[34051]: osdmap e685: 8 total, 8 up, 8 in 2024-03-21T13:26:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:26:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:26:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:26:48.438 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:48 smithi152 ceph-mon[34051]: pgmap v1061: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:26:48.438 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:48.438 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:48.438 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]': finished 2024-03-21T13:26:48.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[30095]: pgmap v1061: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:26:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:48.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]': finished 2024-03-21T13:26:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[39209]: pgmap v1061: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:26:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-136"}]': finished 2024-03-21T13:26:48.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:26:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:26:49.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:49 smithi152 ceph-mon[34051]: osdmap e686: 8 total, 8 up, 8 in 2024-03-21T13:26:49.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]: dispatch 2024-03-21T13:26:49.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]: dispatch 2024-03-21T13:26:49.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:49.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:49 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:49.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]': finished 2024-03-21T13:26:49.575 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:49 smithi152 ceph-mon[34051]: osdmap e687: 8 total, 8 up, 8 in 2024-03-21T13:26:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[30095]: osdmap e686: 8 total, 8 up, 8 in 2024-03-21T13:26:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]: dispatch 2024-03-21T13:26:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]: dispatch 2024-03-21T13:26:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:49.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]': finished 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[30095]: osdmap e687: 8 total, 8 up, 8 in 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[39209]: osdmap e686: 8 total, 8 up, 8 in 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]: dispatch 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]: dispatch 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-136", "mode": "writeback"}]': finished 2024-03-21T13:26:49.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:49 smithi089 ceph-mon[39209]: osdmap e687: 8 total, 8 up, 8 in 2024-03-21T13:26:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:50 smithi152 ceph-mon[34051]: pgmap v1064: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:26:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:50 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:50 smithi089 ceph-mon[30095]: pgmap v1064: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:26:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:50 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:50 smithi089 ceph-mon[39209]: pgmap v1064: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T13:26:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:50.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:50 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:26:51.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[39209]: osdmap e688: 8 total, 8 up, 8 in 2024-03-21T13:26:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:51.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[30095]: osdmap e688: 8 total, 8 up, 8 in 2024-03-21T13:26:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:51.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:26:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:51 smithi152 ceph-mon[34051]: osdmap e688: 8 total, 8 up, 8 in 2024-03-21T13:26:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:51.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]: dispatch 2024-03-21T13:26:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:52 smithi152 ceph-mon[34051]: pgmap v1067: 204 pgs: 204 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-21T13:26:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:52 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]': finished 2024-03-21T13:26:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:52 smithi152 ceph-mon[34051]: osdmap e689: 8 total, 8 up, 8 in 2024-03-21T13:26:52.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[39209]: pgmap v1067: 204 pgs: 204 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-21T13:26:52.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:52.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]': finished 2024-03-21T13:26:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[39209]: osdmap e689: 8 total, 8 up, 8 in 2024-03-21T13:26:52.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[30095]: pgmap v1067: 204 pgs: 204 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-21T13:26:52.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:26:52.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-136"}]': finished 2024-03-21T13:26:52.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:52 smithi089 ceph-mon[30095]: osdmap e689: 8 total, 8 up, 8 in 2024-03-21T13:26:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:53 smithi152 ceph-mon[34051]: osdmap e690: 8 total, 8 up, 8 in 2024-03-21T13:26:53.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:53 smithi089 ceph-mon[39209]: osdmap e690: 8 total, 8 up, 8 in 2024-03-21T13:26:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:53 smithi089 ceph-mon[30095]: osdmap e690: 8 total, 8 up, 8 in 2024-03-21T13:26:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:54 smithi152 ceph-mon[34051]: pgmap v1070: 172 pgs: 172 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-21T13:26:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:54 smithi152 ceph-mon[34051]: osdmap e691: 8 total, 8 up, 8 in 2024-03-21T13:26:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:54.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[39209]: pgmap v1070: 172 pgs: 172 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-21T13:26:54.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[39209]: osdmap e691: 8 total, 8 up, 8 in 2024-03-21T13:26:54.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:54.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[30095]: pgmap v1070: 172 pgs: 172 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-21T13:26:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[30095]: osdmap e691: 8 total, 8 up, 8 in 2024-03-21T13:26:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:54.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:26:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:55 smithi152 ceph-mon[34051]: osdmap e692: 8 total, 8 up, 8 in 2024-03-21T13:26:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:55.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:55.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:55.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[30095]: osdmap e692: 8 total, 8 up, 8 in 2024-03-21T13:26:55.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:55.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:26:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[39209]: osdmap e692: 8 total, 8 up, 8 in 2024-03-21T13:26:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:55.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:26:56.472 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:56 smithi152 ceph-mon[34051]: pgmap v1073: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:56.472 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:56.472 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:56 smithi152 ceph-mon[34051]: osdmap e693: 8 total, 8 up, 8 in 2024-03-21T13:26:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:26:56.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[30095]: pgmap v1073: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[30095]: osdmap e693: 8 total, 8 up, 8 in 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[39209]: pgmap v1073: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:26:56.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[39209]: osdmap e693: 8 total, 8 up, 8 in 2024-03-21T13:26:56.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:26:56.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:26:57.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: osdmap e694: 8 total, 8 up, 8 in 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: osdmap e695: 8 total, 8 up, 8 in 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:26:57.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:26:57.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:26:57.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: osdmap e694: 8 total, 8 up, 8 in 2024-03-21T13:26:57.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:26:57.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:26:57.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: osdmap e695: 8 total, 8 up, 8 in 2024-03-21T13:26:57.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:26:57.498 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: osdmap e694: 8 total, 8 up, 8 in 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: osdmap e695: 8 total, 8 up, 8 in 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:26:57.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-21T13:26:57.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:26:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:26:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:26:58.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:26:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:26:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:58 smithi152 ceph-mon[34051]: pgmap v1076: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:26:59.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:58 smithi152 ceph-mon[34051]: osdmap e696: 8 total, 8 up, 8 in 2024-03-21T13:26:59.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[30095]: pgmap v1076: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[30095]: osdmap e696: 8 total, 8 up, 8 in 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[39209]: pgmap v1076: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-21T13:26:59.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:58 smithi089 ceph-mon[39209]: osdmap e696: 8 total, 8 up, 8 in 2024-03-21T13:27:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]: dispatch 2024-03-21T13:27:00.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:26:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]: dispatch 2024-03-21T13:27:00.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:00.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:00.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:00.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]: dispatch 2024-03-21T13:27:00.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]: dispatch 2024-03-21T13:27:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]: dispatch 2024-03-21T13:27:00.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:26:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]: dispatch 2024-03-21T13:27:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:01 smithi152 ceph-mon[34051]: pgmap v1079: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]': finished 2024-03-21T13:27:01.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:01 smithi152 ceph-mon[34051]: osdmap e697: 8 total, 8 up, 8 in 2024-03-21T13:27:01.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:01 smithi089 ceph-mon[30095]: pgmap v1079: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:01.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]': finished 2024-03-21T13:27:01.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:01 smithi089 ceph-mon[30095]: osdmap e697: 8 total, 8 up, 8 in 2024-03-21T13:27:01.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:01 smithi089 ceph-mon[39209]: pgmap v1079: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:01.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-138"}]': finished 2024-03-21T13:27:01.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:01 smithi089 ceph-mon[39209]: osdmap e697: 8 total, 8 up, 8 in 2024-03-21T13:27:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:02 smithi152 ceph-mon[34051]: osdmap e698: 8 total, 8 up, 8 in 2024-03-21T13:27:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:02 smithi152 ceph-mon[34051]: pgmap v1082: 172 pgs: 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:02 smithi152 ceph-mon[34051]: osdmap e699: 8 total, 8 up, 8 in 2024-03-21T13:27:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:02.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[30095]: osdmap e698: 8 total, 8 up, 8 in 2024-03-21T13:27:02.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[30095]: pgmap v1082: 172 pgs: 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:02.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[30095]: osdmap e699: 8 total, 8 up, 8 in 2024-03-21T13:27:02.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:02.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:02.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[39209]: osdmap e698: 8 total, 8 up, 8 in 2024-03-21T13:27:02.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[39209]: pgmap v1082: 172 pgs: 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:02.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[39209]: osdmap e699: 8 total, 8 up, 8 in 2024-03-21T13:27:02.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:02.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:03 smithi152 ceph-mon[34051]: osdmap e700: 8 total, 8 up, 8 in 2024-03-21T13:27:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:04.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:03 smithi152 ceph-mon[34051]: pgmap v1085: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:04.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:04.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[39209]: osdmap e700: 8 total, 8 up, 8 in 2024-03-21T13:27:04.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:04.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:04.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[39209]: pgmap v1085: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[30095]: osdmap e700: 8 total, 8 up, 8 in 2024-03-21T13:27:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:04.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:03 smithi089 ceph-mon[30095]: pgmap v1085: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:04 smithi152 ceph-mon[34051]: osdmap e701: 8 total, 8 up, 8 in 2024-03-21T13:27:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:27:05.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:27:05.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:05.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[39209]: osdmap e701: 8 total, 8 up, 8 in 2024-03-21T13:27:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:27:05.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:27:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[30095]: osdmap e701: 8 total, 8 up, 8 in 2024-03-21T13:27:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:27:05.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-21T13:27:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]': finished 2024-03-21T13:27:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:05 smithi152 ceph-mon[34051]: osdmap e702: 8 total, 8 up, 8 in 2024-03-21T13:27:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:05 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:27:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:27:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:05 smithi152 ceph-mon[34051]: pgmap v1088: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:27:06.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]': finished 2024-03-21T13:27:06.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]': finished 2024-03-21T13:27:06.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[39209]: osdmap e702: 8 total, 8 up, 8 in 2024-03-21T13:27:06.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:27:06.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:27:06.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[39209]: pgmap v1088: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:27:06.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]': finished 2024-03-21T13:27:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_count","val": "3"}]': finished 2024-03-21T13:27:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[30095]: osdmap e702: 8 total, 8 up, 8 in 2024-03-21T13:27:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:27:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-21T13:27:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[30095]: pgmap v1088: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:27:06.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_period","val": "3"}]': finished 2024-03-21T13:27:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:06 smithi152 ceph-mon[34051]: osdmap e703: 8 total, 8 up, 8 in 2024-03-21T13:27:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:27:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:06 smithi152 ceph-mon[34051]: osdmap e704: 8 total, 8 up, 8 in 2024-03-21T13:27:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:06 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:27:07.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:06 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:27:07.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[30095]: osdmap e703: 8 total, 8 up, 8 in 2024-03-21T13:27:07.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:07.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[30095]: osdmap e704: 8 total, 8 up, 8 in 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[39209]: osdmap e703: 8 total, 8 up, 8 in 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[39209]: osdmap e704: 8 total, 8 up, 8 in 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:27:07.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:06 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-21T13:27:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:27:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:27:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:27:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:07 smithi152 ceph-mon[34051]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:07 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-21T13:27:08.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:07 smithi152 ceph-mon[34051]: osdmap e705: 8 total, 8 up, 8 in 2024-03-21T13:27:08.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:07 smithi089 ceph-mon[30095]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:08.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:07 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-21T13:27:08.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:07 smithi089 ceph-mon[30095]: osdmap e705: 8 total, 8 up, 8 in 2024-03-21T13:27:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:07 smithi089 ceph-mon[39209]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:07 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-21T13:27:08.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:07 smithi089 ceph-mon[39209]: osdmap e705: 8 total, 8 up, 8 in 2024-03-21T13:27:08.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:27:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:27:09.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:08 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:09.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:08 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:09.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:08 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:10.226 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:09 smithi089 ceph-mon[30095]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:10.226 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:09 smithi089 ceph-mon[39209]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:10.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:09 smithi152 ceph-mon[34051]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:12.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:12 smithi089 ceph-mon[39209]: pgmap v1094: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 2 op/s 2024-03-21T13:27:12.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:12 smithi089 ceph-mon[30095]: pgmap v1094: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 2 op/s 2024-03-21T13:27:12.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:12 smithi152 ceph-mon[34051]: pgmap v1094: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 2 op/s 2024-03-21T13:27:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:13.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:13.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:14 smithi152 ceph-mon[34051]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.6 KiB/s wr, 2 op/s 2024-03-21T13:27:14.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:14 smithi089 ceph-mon[30095]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.6 KiB/s wr, 2 op/s 2024-03-21T13:27:14.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:14 smithi089 ceph-mon[39209]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.6 KiB/s wr, 2 op/s 2024-03-21T13:27:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:16 smithi152 ceph-mon[34051]: pgmap v1096: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-21T13:27:16.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:16 smithi089 ceph-mon[39209]: pgmap v1096: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-21T13:27:16.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:16 smithi089 ceph-mon[30095]: pgmap v1096: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-21T13:27:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:27:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:27:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:27:18.465 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:18 smithi152 ceph-mon[34051]: pgmap v1097: 204 pgs: 204 active+clean; 4.5 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 7.4 KiB/s wr, 2 op/s 2024-03-21T13:27:18.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:18 smithi089 ceph-mon[39209]: pgmap v1097: 204 pgs: 204 active+clean; 4.5 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 7.4 KiB/s wr, 2 op/s 2024-03-21T13:27:18.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:18 smithi089 ceph-mon[30095]: pgmap v1097: 204 pgs: 204 active+clean; 4.5 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 7.4 KiB/s wr, 2 op/s 2024-03-21T13:27:18.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:27:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:27:19.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:19.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:20 smithi152 ceph-mon[34051]: pgmap v1098: 204 pgs: 204 active+clean; 4.5 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-03-21T13:27:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:20.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]: dispatch 2024-03-21T13:27:20.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]: dispatch 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[30095]: pgmap v1098: 204 pgs: 204 active+clean; 4.5 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]: dispatch 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]: dispatch 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[39209]: pgmap v1098: 204 pgs: 204 active+clean; 4.5 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:20.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]: dispatch 2024-03-21T13:27:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]: dispatch 2024-03-21T13:27:21.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:21 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]': finished 2024-03-21T13:27:21.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:21 smithi152 ceph-mon[34051]: osdmap e706: 8 total, 8 up, 8 in 2024-03-21T13:27:21.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:21 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]': finished 2024-03-21T13:27:21.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:21 smithi089 ceph-mon[30095]: osdmap e706: 8 total, 8 up, 8 in 2024-03-21T13:27:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:21 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-140"}]': finished 2024-03-21T13:27:21.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:21 smithi089 ceph-mon[39209]: osdmap e706: 8 total, 8 up, 8 in 2024-03-21T13:27:22.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:22 smithi152 ceph-mon[34051]: pgmap v1100: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 7.5 KiB/s wr, 2 op/s 2024-03-21T13:27:22.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:22 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:27:22.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:22 smithi152 ceph-mon[34051]: osdmap e707: 8 total, 8 up, 8 in 2024-03-21T13:27:22.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:22 smithi089 ceph-mon[30095]: pgmap v1100: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 7.5 KiB/s wr, 2 op/s 2024-03-21T13:27:22.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:22 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:27:22.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:22 smithi089 ceph-mon[30095]: osdmap e707: 8 total, 8 up, 8 in 2024-03-21T13:27:22.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:22 smithi089 ceph-mon[39209]: pgmap v1100: 204 pgs: 204 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 7.5 KiB/s wr, 2 op/s 2024-03-21T13:27:22.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:22 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:27:22.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:22 smithi089 ceph-mon[39209]: osdmap e707: 8 total, 8 up, 8 in 2024-03-21T13:27:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:23 smithi152 ceph-mon[34051]: osdmap e708: 8 total, 8 up, 8 in 2024-03-21T13:27:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:23 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:23.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:23 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:23.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:23 smithi089 ceph-mon[30095]: osdmap e708: 8 total, 8 up, 8 in 2024-03-21T13:27:23.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:23 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:23.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:23 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:23.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:23 smithi089 ceph-mon[39209]: osdmap e708: 8 total, 8 up, 8 in 2024-03-21T13:27:23.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:23 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:23.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:23 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:24 smithi152 ceph-mon[34051]: pgmap v1103: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:27:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:24 smithi152 ceph-mon[34051]: osdmap e709: 8 total, 8 up, 8 in 2024-03-21T13:27:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:24.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:24.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[30095]: pgmap v1103: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[30095]: osdmap e709: 8 total, 8 up, 8 in 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[39209]: pgmap v1103: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[39209]: osdmap e709: 8 total, 8 up, 8 in 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:24.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:25 smithi152 ceph-mon[34051]: osdmap e710: 8 total, 8 up, 8 in 2024-03-21T13:27:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:25.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:25.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:25.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[30095]: osdmap e710: 8 total, 8 up, 8 in 2024-03-21T13:27:25.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:25.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[39209]: osdmap e710: 8 total, 8 up, 8 in 2024-03-21T13:27:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:25.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:26 smithi152 ceph-mon[34051]: pgmap v1106: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:27:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]': finished 2024-03-21T13:27:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:26 smithi152 ceph-mon[34051]: osdmap e711: 8 total, 8 up, 8 in 2024-03-21T13:27:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:26 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]: dispatch 2024-03-21T13:27:26.473 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:26 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]: dispatch 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[30095]: pgmap v1106: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]': finished 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[30095]: osdmap e711: 8 total, 8 up, 8 in 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]: dispatch 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]: dispatch 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[39209]: pgmap v1106: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-142"}]': finished 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[39209]: osdmap e711: 8 total, 8 up, 8 in 2024-03-21T13:27:26.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]: dispatch 2024-03-21T13:27:26.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:26 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]: dispatch 2024-03-21T13:27:27.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:27.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]': finished 2024-03-21T13:27:27.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[39209]: osdmap e712: 8 total, 8 up, 8 in 2024-03-21T13:27:27.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:27:27.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:27:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]': finished 2024-03-21T13:27:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[30095]: osdmap e712: 8 total, 8 up, 8 in 2024-03-21T13:27:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:27:27.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:27:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:27 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-142", "mode": "writeback"}]': finished 2024-03-21T13:27:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:27 smithi152 ceph-mon[34051]: osdmap e712: 8 total, 8 up, 8 in 2024-03-21T13:27:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:27:27.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:27:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:27:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:27:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:27:28.474 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-mon[34051]: pgmap v1109: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:27:28.474 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:27:28.474 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-mon[34051]: osdmap e713: 8 total, 8 up, 8 in 2024-03-21T13:27:28.474 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:27:28.474 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:27:28.474 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:28.475 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[30095]: pgmap v1109: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[30095]: osdmap e713: 8 total, 8 up, 8 in 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:28.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[39209]: pgmap v1109: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:27:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:27:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[39209]: osdmap e713: 8 total, 8 up, 8 in 2024-03-21T13:27:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:27:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:27:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:28.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:28.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:27:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:27:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:27:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:29 smithi152 ceph-mon[34051]: osdmap e714: 8 total, 8 up, 8 in 2024-03-21T13:27:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:29.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:29.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:27:29.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[30095]: osdmap e714: 8 total, 8 up, 8 in 2024-03-21T13:27:29.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:29.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:27:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[39209]: osdmap e714: 8 total, 8 up, 8 in 2024-03-21T13:27:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:27:29.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:30 smithi152 ceph-mon[34051]: pgmap v1112: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:27:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:30 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:27:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:27:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:30 smithi152 ceph-mon[34051]: osdmap e715: 8 total, 8 up, 8 in 2024-03-21T13:27:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:27:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:27:30.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:30 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:27:30.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[30095]: pgmap v1112: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:27:30.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[30095]: osdmap e715: 8 total, 8 up, 8 in 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[39209]: pgmap v1112: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[39209]: osdmap e715: 8 total, 8 up, 8 in 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:27:30.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:27:30.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:30 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:27:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:27:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:31 smithi152 ceph-mon[34051]: osdmap e716: 8 total, 8 up, 8 in 2024-03-21T13:27:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:27:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[39209]: osdmap e716: 8 total, 8 up, 8 in 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[30095]: osdmap e716: 8 total, 8 up, 8 in 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:27:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-21T13:27:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:32 smithi152 ceph-mon[34051]: pgmap v1115: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-21T13:27:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:32 smithi152 ceph-mon[34051]: osdmap e717: 8 total, 8 up, 8 in 2024-03-21T13:27:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:32 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:27:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:32 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:27:32.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[30095]: pgmap v1115: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[30095]: osdmap e717: 8 total, 8 up, 8 in 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[39209]: pgmap v1115: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[39209]: osdmap e717: 8 total, 8 up, 8 in 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:27:32.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:32 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-21T13:27:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-21T13:27:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:33 smithi152 ceph-mon[34051]: osdmap e718: 8 total, 8 up, 8 in 2024-03-21T13:27:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-21T13:27:33.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:33 smithi089 ceph-mon[30095]: osdmap e718: 8 total, 8 up, 8 in 2024-03-21T13:27:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-21T13:27:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:33 smithi089 ceph-mon[39209]: osdmap e718: 8 total, 8 up, 8 in 2024-03-21T13:27:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:34 smithi152 ceph-mon[34051]: pgmap v1118: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:34.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:34 smithi089 ceph-mon[39209]: pgmap v1118: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:34 smithi089 ceph-mon[30095]: pgmap v1118: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:27:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:35 smithi152 ceph-mon[34051]: osdmap e719: 8 total, 8 up, 8 in 2024-03-21T13:27:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:35.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:27:35.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[30095]: osdmap e719: 8 total, 8 up, 8 in 2024-03-21T13:27:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:27:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[39209]: osdmap e719: 8 total, 8 up, 8 in 2024-03-21T13:27:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:36 smithi152 ceph-mon[34051]: pgmap v1120: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:27:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]': finished 2024-03-21T13:27:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:36 smithi152 ceph-mon[34051]: osdmap e720: 8 total, 8 up, 8 in 2024-03-21T13:27:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:36.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:36 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[30095]: pgmap v1120: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:27:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]': finished 2024-03-21T13:27:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[30095]: osdmap e720: 8 total, 8 up, 8 in 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[39209]: pgmap v1120: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]': finished 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[39209]: osdmap e720: 8 total, 8 up, 8 in 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:36 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-142"}]: dispatch 2024-03-21T13:27:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:37 smithi089 ceph-mon[30095]: osdmap e721: 8 total, 8 up, 8 in 2024-03-21T13:27:37.497 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:37 smithi089 ceph-mon[39209]: osdmap e721: 8 total, 8 up, 8 in 2024-03-21T13:27:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:37 smithi152 ceph-mon[34051]: osdmap e721: 8 total, 8 up, 8 in 2024-03-21T13:27:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:27:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:27:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:27:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:38 smithi152 ceph-mon[34051]: pgmap v1123: 172 pgs: 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:27:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:38 smithi152 ceph-mon[34051]: osdmap e722: 8 total, 8 up, 8 in 2024-03-21T13:27:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:38.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:38.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:27:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[30095]: pgmap v1123: 172 pgs: 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[30095]: osdmap e722: 8 total, 8 up, 8 in 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[39209]: pgmap v1123: 172 pgs: 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[39209]: osdmap e722: 8 total, 8 up, 8 in 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:39 smithi152 ceph-mon[34051]: osdmap e723: 8 total, 8 up, 8 in 2024-03-21T13:27:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[30095]: osdmap e723: 8 total, 8 up, 8 in 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[39209]: osdmap e723: 8 total, 8 up, 8 in 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:40.573 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:40 smithi152 ceph-mon[34051]: pgmap v1126: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:27:40.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:40.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:40 smithi152 ceph-mon[34051]: osdmap e724: 8 total, 8 up, 8 in 2024-03-21T13:27:40.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:40 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:40.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:40.574 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:40 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[39209]: pgmap v1126: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[39209]: osdmap e724: 8 total, 8 up, 8 in 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[30095]: pgmap v1126: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[30095]: osdmap e724: 8 total, 8 up, 8 in 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:40.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:40.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:40 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]': finished 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: osdmap e725: 8 total, 8 up, 8 in 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]: dispatch 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]: dispatch 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:41 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]': finished 2024-03-21T13:27:41.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: osdmap e725: 8 total, 8 up, 8 in 2024-03-21T13:27:41.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]: dispatch 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]: dispatch 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-144"}]': finished 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: osdmap e725: 8 total, 8 up, 8 in 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]: dispatch 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]: dispatch 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:41.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:41 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:42 smithi152 ceph-mon[34051]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:27:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:42 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:42 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]': finished 2024-03-21T13:27:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:42 smithi152 ceph-mon[34051]: osdmap e726: 8 total, 8 up, 8 in 2024-03-21T13:27:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:27:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:27:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:42 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:42.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[30095]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:27:42.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]': finished 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[30095]: osdmap e726: 8 total, 8 up, 8 in 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[39209]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-144", "mode": "readproxy"}]': finished 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[39209]: osdmap e726: 8 total, 8 up, 8 in 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:27:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:42 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:43.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:43.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:44 smithi152 ceph-mon[34051]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 213 B/s wr, 1 op/s 2024-03-21T13:27:44.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:44 smithi089 ceph-mon[30095]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 213 B/s wr, 1 op/s 2024-03-21T13:27:44.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:44 smithi089 ceph-mon[39209]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 213 B/s wr, 1 op/s 2024-03-21T13:27:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:46 smithi152 ceph-mon[34051]: pgmap v1132: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-21T13:27:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:46 smithi089 ceph-mon[30095]: pgmap v1132: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-21T13:27:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:46 smithi089 ceph-mon[39209]: pgmap v1132: 204 pgs: 204 active+clean; 4.4 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-21T13:27:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:27:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:27:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:27:48.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:48 smithi152 ceph-mon[34051]: pgmap v1133: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 131 B/s wr, 1 op/s 2024-03-21T13:27:48.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:27:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:27:48.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:48 smithi089 ceph-mon[30095]: pgmap v1133: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 131 B/s wr, 1 op/s 2024-03-21T13:27:48.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:48 smithi089 ceph-mon[39209]: pgmap v1133: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 131 B/s wr, 1 op/s 2024-03-21T13:27:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:49.571 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:27:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=sqlstore.transactions t=2024-03-21T13:27:49.284381824Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T13:27:49.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:49.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:50 smithi152 ceph-mon[34051]: pgmap v1134: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 116 B/s wr, 1 op/s 2024-03-21T13:27:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:50 smithi089 ceph-mon[30095]: pgmap v1134: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 116 B/s wr, 1 op/s 2024-03-21T13:27:50.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:50 smithi089 ceph-mon[39209]: pgmap v1134: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 116 B/s wr, 1 op/s 2024-03-21T13:27:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:52 smithi152 ceph-mon[34051]: pgmap v1135: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:27:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:52.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:52 smithi089 ceph-mon[30095]: pgmap v1135: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:27:52.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:52.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:52 smithi089 ceph-mon[39209]: pgmap v1135: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:27:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:27:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:53 smithi152 ceph-mon[34051]: osdmap e727: 8 total, 8 up, 8 in 2024-03-21T13:27:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:53.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:27:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[30095]: osdmap e727: 8 total, 8 up, 8 in 2024-03-21T13:27:53.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:53.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:27:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[39209]: osdmap e727: 8 total, 8 up, 8 in 2024-03-21T13:27:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:53.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: pgmap v1137: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:27:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:27:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]': finished 2024-03-21T13:27:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: osdmap e728: 8 total, 8 up, 8 in 2024-03-21T13:27:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:54.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:54.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: pgmap v1137: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:27:54.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:27:54.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]': finished 2024-03-21T13:27:54.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: osdmap e728: 8 total, 8 up, 8 in 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: pgmap v1137: 204 pgs: 204 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]': finished 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: osdmap e728: 8 total, 8 up, 8 in 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:27:54.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:54.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-144"}]: dispatch 2024-03-21T13:27:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:55 smithi152 ceph-mon[34051]: osdmap e729: 8 total, 8 up, 8 in 2024-03-21T13:27:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:55 smithi152 ceph-mon[34051]: osdmap e730: 8 total, 8 up, 8 in 2024-03-21T13:27:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:55.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[30095]: osdmap e729: 8 total, 8 up, 8 in 2024-03-21T13:27:55.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[30095]: osdmap e730: 8 total, 8 up, 8 in 2024-03-21T13:27:55.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:55.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:55.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[39209]: osdmap e729: 8 total, 8 up, 8 in 2024-03-21T13:27:55.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[39209]: osdmap e730: 8 total, 8 up, 8 in 2024-03-21T13:27:55.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:55.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[39209]: pgmap v1140: 172 pgs: 172 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[39209]: osdmap e731: 8 total, 8 up, 8 in 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[30095]: pgmap v1140: 172 pgs: 172 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:56.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[30095]: osdmap e731: 8 total, 8 up, 8 in 2024-03-21T13:27:56.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:56.826 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:56 smithi152 ceph-mon[34051]: pgmap v1140: 172 pgs: 172 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:27:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:27:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:56 smithi152 ceph-mon[34051]: osdmap e731: 8 total, 8 up, 8 in 2024-03-21T13:27:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:57.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:27:57.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:27:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:27:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:27:58.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:27:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-mon[34051]: pgmap v1143: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-mon[34051]: osdmap e732: 8 total, 8 up, 8 in 2024-03-21T13:27:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:27:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:27:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:58.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:58.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[30095]: pgmap v1143: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[30095]: osdmap e732: 8 total, 8 up, 8 in 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[39209]: pgmap v1143: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[39209]: osdmap e732: 8 total, 8 up, 8 in 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:27:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]': finished 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: osdmap e733: 8 total, 8 up, 8 in 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]: dispatch 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]: dispatch 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]': finished 2024-03-21T13:27:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:27:59 smithi152 ceph-mon[34051]: osdmap e734: 8 total, 8 up, 8 in 2024-03-21T13:27:59.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]': finished 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: osdmap e733: 8 total, 8 up, 8 in 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]: dispatch 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]: dispatch 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]': finished 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[30095]: osdmap e734: 8 total, 8 up, 8 in 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi089-79633-111", "overlaypool": "test-rados-api-smithi089-79633-146"}]': finished 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: osdmap e733: 8 total, 8 up, 8 in 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]: dispatch 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]: dispatch 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:27:59.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-21T13:27:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi089-79633-146", "mode": "writeback"}]': finished 2024-03-21T13:27:59.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:27:59 smithi089 ceph-mon[39209]: osdmap e734: 8 total, 8 up, 8 in 2024-03-21T13:28:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:00 smithi152 ceph-mon[34051]: pgmap v1146: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:00 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:28:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:00 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:28:00.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:00 smithi089 ceph-mon[30095]: pgmap v1146: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:00.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:00 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:28:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:00 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:28:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:00 smithi089 ceph-mon[39209]: pgmap v1146: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:00 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:28:00.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:00 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-21T13:28:01.797 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:28:01.797 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:01 smithi152 ceph-mon[34051]: osdmap e735: 8 total, 8 up, 8 in 2024-03-21T13:28:01.798 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:01 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:28:01.798 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:01 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:28:01.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:28:01.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[30095]: osdmap e735: 8 total, 8 up, 8 in 2024-03-21T13:28:01.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:28:01.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:28:01.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_count","val": "2"}]': finished 2024-03-21T13:28:01.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[39209]: osdmap e735: 8 total, 8 up, 8 in 2024-03-21T13:28:01.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:28:01.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:01 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-21T13:28:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:02 smithi152 ceph-mon[34051]: pgmap v1149: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:28:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:28:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:02 smithi152 ceph-mon[34051]: osdmap e736: 8 total, 8 up, 8 in 2024-03-21T13:28:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:02 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:28:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:02 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:28:02.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[39209]: pgmap v1149: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:28:02.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[39209]: osdmap e736: 8 total, 8 up, 8 in 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[30095]: pgmap v1149: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_period","val": "600"}]': finished 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[30095]: osdmap e736: 8 total, 8 up, 8 in 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:28:02.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:02 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-21T13:28:03.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:03 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:28:03.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:28:03.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:03 smithi152 ceph-mon[34051]: osdmap e737: 8 total, 8 up, 8 in 2024-03-21T13:28:03.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:03 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:28:03.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:03 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:28:03.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:28:03.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[30095]: osdmap e737: 8 total, 8 up, 8 in 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[39209]: osdmap e737: 8 total, 8 up, 8 in 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:28:03.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:03 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-21T13:28:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:04 smithi152 ceph-mon[34051]: pgmap v1152: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:28:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:28:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:04 smithi152 ceph-mon[34051]: osdmap e738: 8 total, 8 up, 8 in 2024-03-21T13:28:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:04 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:28:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:04 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:28:04.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[30095]: pgmap v1152: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[30095]: osdmap e738: 8 total, 8 up, 8 in 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[39209]: pgmap v1152: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[39209]: osdmap e738: 8 total, 8 up, 8 in 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:28:04.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:04 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-21T13:28:05.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:05 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]': finished 2024-03-21T13:28:05.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:05 smithi152 ceph-mon[34051]: osdmap e739: 8 total, 8 up, 8 in 2024-03-21T13:28:05.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:05 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]': finished 2024-03-21T13:28:05.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:05 smithi089 ceph-mon[30095]: osdmap e739: 8 total, 8 up, 8 in 2024-03-21T13:28:05.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:05 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-146","var": "target_max_objects","val": "1"}]': finished 2024-03-21T13:28:05.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:05 smithi089 ceph-mon[39209]: osdmap e739: 8 total, 8 up, 8 in 2024-03-21T13:28:06.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:06 smithi152 ceph-mon[34051]: pgmap v1155: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:06.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:06 smithi152 ceph-mon[34051]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-21T13:28:06.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:06 smithi089 ceph-mon[30095]: pgmap v1155: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:06.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:06 smithi089 ceph-mon[30095]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-21T13:28:06.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:06 smithi089 ceph-mon[39209]: pgmap v1155: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:06.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:06 smithi089 ceph-mon[39209]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-21T13:28:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:28:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:28:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:28:08.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:08 smithi152 ceph-mon[34051]: pgmap v1156: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:08.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:28:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:28:08.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:08 smithi089 ceph-mon[30095]: pgmap v1156: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:08.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:08 smithi089 ceph-mon[39209]: pgmap v1156: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:09.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:09.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:10.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:10 smithi089 ceph-mon[30095]: pgmap v1157: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:10.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:10 smithi089 ceph-mon[39209]: pgmap v1157: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:10.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:10 smithi152 ceph-mon[34051]: pgmap v1157: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:12 smithi152 ceph-mon[34051]: pgmap v1158: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:12.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:12 smithi089 ceph-mon[30095]: pgmap v1158: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:12.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:12 smithi089 ceph-mon[39209]: pgmap v1158: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:14.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:13 smithi152 ceph-mon[34051]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:14.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:14.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:14.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:13 smithi089 ceph-mon[30095]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:14.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:13 smithi089 ceph-mon[39209]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:15.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:14 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:15.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:15.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:14 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:15.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:14 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:15 smithi152 ceph-mon[34051]: osdmap e740: 8 total, 8 up, 8 in 2024-03-21T13:28:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:15 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:15 smithi152 ceph-mon[34051]: pgmap v1161: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:15 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]': finished 2024-03-21T13:28:16.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:15 smithi152 ceph-mon[34051]: osdmap e741: 8 total, 8 up, 8 in 2024-03-21T13:28:16.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:16.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[30095]: osdmap e740: 8 total, 8 up, 8 in 2024-03-21T13:28:16.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:16.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[30095]: pgmap v1161: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]': finished 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[30095]: osdmap e741: 8 total, 8 up, 8 in 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[39209]: osdmap e740: 8 total, 8 up, 8 in 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[39209]: pgmap v1161: 204 pgs: 204 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]': finished 2024-03-21T13:28:16.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:15 smithi089 ceph-mon[39209]: osdmap e741: 8 total, 8 up, 8 in 2024-03-21T13:28:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:16 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:17.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:16 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:17.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:17.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:17.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:17.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:17.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:16 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-146"}]: dispatch 2024-03-21T13:28:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:28:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:28:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:28:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:17 smithi152 ceph-mon[34051]: osdmap e742: 8 total, 8 up, 8 in 2024-03-21T13:28:18.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:17 smithi152 ceph-mon[34051]: pgmap v1164: 172 pgs: 172 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T13:28:18.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:17 smithi089 ceph-mon[30095]: osdmap e742: 8 total, 8 up, 8 in 2024-03-21T13:28:18.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:17 smithi089 ceph-mon[30095]: pgmap v1164: 172 pgs: 172 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T13:28:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:17 smithi089 ceph-mon[39209]: osdmap e742: 8 total, 8 up, 8 in 2024-03-21T13:28:18.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:17 smithi089 ceph-mon[39209]: pgmap v1164: 172 pgs: 172 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T13:28:18.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:28:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:28:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:18 smithi152 ceph-mon[34051]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-21T13:28:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:18 smithi152 ceph-mon[34051]: osdmap e743: 8 total, 8 up, 8 in 2024-03-21T13:28:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:18 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:18 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:19.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:18 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[30095]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[30095]: osdmap e743: 8 total, 8 up, 8 in 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[39209]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[39209]: osdmap e743: 8 total, 8 up, 8 in 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:19.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:18 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:19 smithi152 ceph-mon[34051]: osdmap e744: 8 total, 8 up, 8 in 2024-03-21T13:28:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:19 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:28:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:19 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:28:20.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:19 smithi152 ceph-mon[34051]: pgmap v1167: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[30095]: osdmap e744: 8 total, 8 up, 8 in 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[30095]: pgmap v1167: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[39209]: osdmap e744: 8 total, 8 up, 8 in 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-21T13:28:20.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:19 smithi089 ceph-mon[39209]: pgmap v1167: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T13:28:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:28:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:20 smithi152 ceph-mon[34051]: osdmap e745: 8 total, 8 up, 8 in 2024-03-21T13:28:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:20 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:21.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:20 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:21.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:28:21.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[30095]: osdmap e745: 8 total, 8 up, 8 in 2024-03-21T13:28:21.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:21.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-21T13:28:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[39209]: osdmap e745: 8 total, 8 up, 8 in 2024-03-21T13:28:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:21.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:20 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]': finished 2024-03-21T13:28:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:22 smithi152 ceph-mon[34051]: osdmap e746: 8 total, 8 up, 8 in 2024-03-21T13:28:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:22.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:22 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:22 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:22.322 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:22 smithi152 ceph-mon[34051]: pgmap v1170: 204 pgs: 204 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:28:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]': finished 2024-03-21T13:28:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[30095]: osdmap e746: 8 total, 8 up, 8 in 2024-03-21T13:28:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:22.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[30095]: pgmap v1170: 204 pgs: 204 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:28:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]': finished 2024-03-21T13:28:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[39209]: osdmap e746: 8 total, 8 up, 8 in 2024-03-21T13:28:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-148"}]: dispatch 2024-03-21T13:28:22.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:22 smithi089 ceph-mon[39209]: pgmap v1170: 204 pgs: 204 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-21T13:28:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:23 smithi152 ceph-mon[34051]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:23.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:23 smithi152 ceph-mon[34051]: osdmap e747: 8 total, 8 up, 8 in 2024-03-21T13:28:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:23 smithi089 ceph-mon[30095]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:23.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:23 smithi089 ceph-mon[30095]: osdmap e747: 8 total, 8 up, 8 in 2024-03-21T13:28:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:23 smithi089 ceph-mon[39209]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:23.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:23 smithi089 ceph-mon[39209]: osdmap e747: 8 total, 8 up, 8 in 2024-03-21T13:28:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:24 smithi152 ceph-mon[34051]: osdmap e748: 8 total, 8 up, 8 in 2024-03-21T13:28:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:24 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:24 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:24.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:24 smithi152 ceph-mon[34051]: pgmap v1173: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:28:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[30095]: osdmap e748: 8 total, 8 up, 8 in 2024-03-21T13:28:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:24.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[30095]: pgmap v1173: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:28:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[39209]: osdmap e748: 8 total, 8 up, 8 in 2024-03-21T13:28:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:24.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:24 smithi089 ceph-mon[39209]: pgmap v1173: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-21T13:28:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:25 smithi152 ceph-mon[34051]: osdmap e749: 8 total, 8 up, 8 in 2024-03-21T13:28:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:25 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-150"}]: dispatch 2024-03-21T13:28:25.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:25 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-150"}]: dispatch 2024-03-21T13:28:25.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:25.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[30095]: osdmap e749: 8 total, 8 up, 8 in 2024-03-21T13:28:25.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:25.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-150"}]: dispatch 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-150"}]: dispatch 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[39209]: osdmap e749: 8 total, 8 up, 8 in 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-150"}]: dispatch 2024-03-21T13:28:25.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:25 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-150"}]: dispatch 2024-03-21T13:28:26.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:26 smithi152 ceph-mon[34051]: osdmap e750: 8 total, 8 up, 8 in 2024-03-21T13:28:26.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:26 smithi152 ceph-mon[34051]: pgmap v1176: 172 pgs: 172 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:26.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:26 smithi089 ceph-mon[39209]: osdmap e750: 8 total, 8 up, 8 in 2024-03-21T13:28:26.324 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:26 smithi089 ceph-mon[39209]: pgmap v1176: 172 pgs: 172 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:26 smithi089 ceph-mon[30095]: osdmap e750: 8 total, 8 up, 8 in 2024-03-21T13:28:26.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:26 smithi089 ceph-mon[30095]: pgmap v1176: 172 pgs: 172 active+clean; 4.4 MiB data, 797 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:27 smithi152 ceph-mon[34051]: osdmap e751: 8 total, 8 up, 8 in 2024-03-21T13:28:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:27.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:27 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:27.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:27 smithi089 ceph-mon[30095]: osdmap e751: 8 total, 8 up, 8 in 2024-03-21T13:28:27.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:27.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:27 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:27 smithi089 ceph-mon[39209]: osdmap e751: 8 total, 8 up, 8 in 2024-03-21T13:28:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:27.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:27 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:28:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:28:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: osdmap e752: 8 total, 8 up, 8 in 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: pgmap v1179: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-152"}]: dispatch 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-152"}]: dispatch 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:28.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:28.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: osdmap e752: 8 total, 8 up, 8 in 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: pgmap v1179: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-152"}]: dispatch 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-152"}]: dispatch 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: osdmap e752: 8 total, 8 up, 8 in 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: pgmap v1179: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:28.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-152"}]: dispatch 2024-03-21T13:28:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-152"}]: dispatch 2024-03-21T13:28:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:28.326 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:28.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:28:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:28:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:29 smithi152 ceph-mon[34051]: osdmap e753: 8 total, 8 up, 8 in 2024-03-21T13:28:29.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:29.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:29 smithi089 ceph-mon[30095]: osdmap e753: 8 total, 8 up, 8 in 2024-03-21T13:28:29.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:29 smithi089 ceph-mon[39209]: osdmap e753: 8 total, 8 up, 8 in 2024-03-21T13:28:29.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:30 smithi152 ceph-mon[34051]: pgmap v1181: 172 pgs: 172 active+clean; 4.4 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 254 B/s wr, 2 op/s 2024-03-21T13:28:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:30 smithi152 ceph-mon[34051]: osdmap e754: 8 total, 8 up, 8 in 2024-03-21T13:28:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:30 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:30.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:30 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:30.324 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[30095]: pgmap v1181: 172 pgs: 172 active+clean; 4.4 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 254 B/s wr, 2 op/s 2024-03-21T13:28:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[30095]: osdmap e754: 8 total, 8 up, 8 in 2024-03-21T13:28:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:30.325 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[39209]: pgmap v1181: 172 pgs: 172 active+clean; 4.4 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 254 B/s wr, 2 op/s 2024-03-21T13:28:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[39209]: osdmap e754: 8 total, 8 up, 8 in 2024-03-21T13:28:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:30.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:30 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-21T13:28:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: osdmap e755: 8 total, 8 up, 8 in 2024-03-21T13:28:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-111","var": "dedup_tier","val": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-111","var": "dedup_tier","val": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.572 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:31 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: osdmap e755: 8 total, 8 up, 8 in 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-111","var": "dedup_tier","val": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-111","var": "dedup_tier","val": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi089-79633-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: osdmap e755: 8 total, 8 up, 8 in 2024-03-21T13:28:31.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-111","var": "dedup_tier","val": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi089-79633-111","var": "dedup_tier","val": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi089-79633-111", "tierpool": "test-rados-api-smithi089-79633-154"}]: dispatch 2024-03-21T13:28:31.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:31 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:32 smithi152 ceph-mon[34051]: pgmap v1184: 204 pgs: 204 active+clean; 4.4 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-21T13:28:32.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:32 smithi152 ceph-mon[34051]: osdmap e756: 8 total, 8 up, 8 in 2024-03-21T13:28:32.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:32 smithi089 ceph-mon[39209]: pgmap v1184: 204 pgs: 204 active+clean; 4.4 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-21T13:28:32.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:32 smithi089 ceph-mon[39209]: osdmap e756: 8 total, 8 up, 8 in 2024-03-21T13:28:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:32 smithi089 ceph-mon[30095]: pgmap v1184: 204 pgs: 204 active+clean; 4.4 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-21T13:28:32.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:32 smithi089 ceph-mon[30095]: osdmap e756: 8 total, 8 up, 8 in 2024-03-21T13:28:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:33 smithi152 ceph-mon[34051]: osdmap e757: 8 total, 8 up, 8 in 2024-03-21T13:28:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:33 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:33.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:33 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:33 smithi089 ceph-mon[30095]: osdmap e757: 8 total, 8 up, 8 in 2024-03-21T13:28:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:33 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:33.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:33 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:33 smithi089 ceph-mon[39209]: osdmap e757: 8 total, 8 up, 8 in 2024-03-21T13:28:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:33 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:33.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:33 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:34.114 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7899 ms) 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12769 ms) 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8064 ms) 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13093 ms) 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7095 ms) 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8068 ms) 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-21T13:28:34.115 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: ok, hit_set contains 337:602f83fe:::foo:head 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8717 ms) 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: first is 1711027628 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,1711027633,1711027634,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,1711027633,1711027634,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,1711027633,1711027634,0 2024-03-21T13:28:34.116 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,1711027633,1711027634,1711027636,1711027637,0 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,1711027633,1711027634,1711027636,1711027637,0 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027628,1711027630,1711027631,1711027633,1711027634,1711027636,1711027637,0 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: got ls 1711027631,1711027633,1711027634,1711027636,1711027637,1711027639,1711027640,0 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: first now 1711027631, trimmed 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20261 ms) 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: foo0 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (15101 ms) 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18095 ms) 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22649 ms) 2024-03-21T13:28:34.117 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5047 ms) 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3011 ms) 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3030 ms) 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3028 ms) 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (230574 ms total) 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (861866 ms total) 2024-03-21T13:28:34.118 INFO:tasks.workunit.client.0.smithi089.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-21T13:28:34.124 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79398 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79398 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79577 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79577 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80147 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80147 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79827 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79827 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79713 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79713 2024-03-21T13:28:34.125 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79468 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79468 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79765 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79765 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80312 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80312 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80317 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80317 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79489 2024-03-21T13:28:34.126 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79489 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80200 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80200 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80060 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80060 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79371 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79371 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79435 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79435 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80091 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80091 2024-03-21T13:28:34.127 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79362 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79362 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79382 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79382 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80481 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80481 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79655 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79655 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80593 2024-03-21T13:28:34.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80593 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79752 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79752 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80263 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80263 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80654 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80654 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79679 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79679 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.129 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80111 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80111 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80431 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80431 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80221 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80221 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80518 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80518 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80025 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80025 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.130 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79531 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79531 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80629 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80629 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79869 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79869 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79355 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79355 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79415 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79415 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79792 2024-03-21T13:28:34.131 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79792 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79455 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79455 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79915 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79915 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=79967 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 79967 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80361 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80361 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ for t in "${!pids[@]}" 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=80671 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 80671 2024-03-21T13:28:34.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ exit 0 2024-03-21T13:28:34.133 INFO:tasks.workunit.client.0.smithi089.stderr:+ cleanup 2024-03-21T13:28:34.133 INFO:tasks.workunit.client.0.smithi089.stderr:+ pkill -P 79349 2024-03-21T13:28:34.143 INFO:tasks.workunit.client.0.smithi089.stderr:+ true 2024-03-21T13:28:34.143 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-21T13:28:34.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-21T13:28:34.176 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-21T13:28:34.177 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-21T13:28:34.177 DEBUG:teuthology.orchestra.run.smithi089:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=960fbbe4645c2becc8084669be194c95183c7753 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-21T13:28:34.236 INFO:tasks.workunit.client.0.smithi089.stderr:++ uuidgen 2024-03-21T13:28:34.242 INFO:tasks.workunit.client.0.smithi089.stderr:+ p=f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:28:34.242 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool create f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 12 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.324+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3982267015 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 msgr2=0x7fcbf8107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.324+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3982267015 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf8107a50 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009c10 tx=0x7fcbe802bef0 comp rx=0 tx=0).stop 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.324+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3982267015 shutdown_connections 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.324+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3982267015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbf8102020 0x7fcbf810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.324+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3982267015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 0x7fcbf8101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.324+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3982267015 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf8107a50 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.324+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3982267015 >> 172.21.15.89:0/3982267015 conn(0x7fcbf80fd130 msgr2=0x7fcbf80ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.325+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3982267015 shutdown_connections 2024-03-21T13:28:34.325 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.325+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3982267015 wait complete. 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.325+0000 7fcbfd35e640 1 Processor -- start 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.325+0000 7fcbfd35e640 1 -- start start 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 0x7fcbf8197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbf8102020 0x7fcbf8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf819bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8110f00 con 0x7fcbf8101660 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8110d80 con 0x7fcbf8102020 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8111080 con 0x7fcbf8107670 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 0x7fcbf8197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:34.328 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf77fe640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf819bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 0x7fcbf8197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59532/0 (socket says 172.21.15.89:59532) 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 -- 172.21.15.89:0/3199288474 learned_addr learned my addr 172.21.15.89:0/3199288474 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf77fe640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf819bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:41496/0 (socket says 172.21.15.89:41496) 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf67fc640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbf8102020 0x7fcbf8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 -- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 msgr2=0x7fcbf819bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf819bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 -- 172.21.15.89:0/3199288474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbf8102020 msgr2=0x7fcbf8197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbf8102020 0x7fcbf8197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 -- 172.21.15.89:0/3199288474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf819c500 con 0x7fcbf8101660 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf77fe640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf819bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf67fc640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbf8102020 0x7fcbf8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbf6ffd640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 0x7fcbf8197510 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80096f0 tx=0x7fcbe8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8043020 con 0x7fcbf8101660 2024-03-21T13:28:34.329 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf819c790 con 0x7fcbf8101660 2024-03-21T13:28:34.330 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcbe8003e10 con 0x7fcbf8101660 2024-03-21T13:28:34.330 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8034430 con 0x7fcbf8101660 2024-03-21T13:28:34.330 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.326+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf80738d0 con 0x7fcbf8101660 2024-03-21T13:28:34.330 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.327+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf8068580 con 0x7fcbf8101660 2024-03-21T13:28:34.330 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.328+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fcbe8046020 con 0x7fcbf8101660 2024-03-21T13:28:34.331 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.328+0000 7fcbdffff640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcbe006e2b0 0x7fcbe0070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:34.331 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.329+0000 7fcbf67fc640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcbe006e2b0 0x7fcbe0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:34.331 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.329+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(759..759 src has 1..759) v4 ==== 8775+0+0 (secure 0 0 0) 0x7fcbe80b13e0 con 0x7fcbf8101660 2024-03-21T13:28:34.331 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.329+0000 7fcbf67fc640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcbe006e2b0 0x7fcbe0070770 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80ff8d0 tx=0x7fcbec009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:28:34.334 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.334+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcbe804d020 con 0x7fcbf8101660 2024-03-21T13:28:34.513 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:34.513+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12} v 0) v1 -- 0x7fcbf8109bd0 con 0x7fcbf8101660 2024-03-21T13:28:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:34 smithi152 ceph-mon[34051]: pgmap v1187: 164 pgs: 164 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:34 smithi152 ceph-mon[34051]: osdmap e758: 8 total, 8 up, 8 in 2024-03-21T13:28:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:34 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:34.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:34 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[30095]: pgmap v1187: 164 pgs: 164 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:34.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[30095]: osdmap e758: 8 total, 8 up, 8 in 2024-03-21T13:28:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:34.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[39209]: pgmap v1187: 164 pgs: 164 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[39209]: osdmap e758: 8 total, 8 up, 8 in 2024-03-21T13:28:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199150366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:34.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:34 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]: dispatch 2024-03-21T13:28:35.110 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.110+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]=0 pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' created v760) v1 ==== 176+0+0 (secure 0 0 0) 0x7fcbe807e110 con 0x7fcbf8101660 2024-03-21T13:28:35.220 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.219+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12} v 0) v1 -- 0x7fcbf8109760 con 0x7fcbf8101660 2024-03-21T13:28:35.220 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.219+0000 7fcbdffff640 1 -- 172.21.15.89:0/3199288474 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]=0 pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' already exists v760) v1 ==== 183+0+0 (secure 0 0 0) 0x7fcbe8081b60 con 0x7fcbf8101660 2024-03-21T13:28:35.220 INFO:tasks.workunit.client.0.smithi089.stderr:pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' already exists 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcbe006e2b0 msgr2=0x7fcbe0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcbe006e2b0 0x7fcbe0070770 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80ff8d0 tx=0x7fcbec009260 comp rx=0 tx=0).stop 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 msgr2=0x7fcbf8197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 0x7fcbf8197510 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80096f0 tx=0x7fcbe8003c20 comp rx=0 tx=0).stop 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 shutdown_connections 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcbe006e2b0 0x7fcbe0070770 unknown :-1 s=CLOSED pgs=3449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbf8107670 0x7fcbf819bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcbf8102020 0x7fcbf8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 --2- 172.21.15.89:0/3199288474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8101660 0x7fcbf8197510 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.222 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.221+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 >> 172.21.15.89:0/3199288474 conn(0x7fcbf80fd130 msgr2=0x7fcbf80fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:28:35.223 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.222+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 shutdown_connections 2024-03-21T13:28:35.223 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.222+0000 7fcbfd35e640 1 -- 172.21.15.89:0/3199288474 wait complete. 2024-03-21T13:28:35.231 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a max_objects 10 2024-03-21T13:28:35.316 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 -- 172.21.15.89:0/994418194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a881078a0 msgr2=0x7f0a88107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/994418194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a881078a0 0x7f0a88107c80 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009930 tx=0x7f0a8402bcd0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 -- 172.21.15.89:0/994418194 shutdown_connections 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/994418194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a88100920 0x7f0a8810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/994418194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a880fff60 0x7f0a881003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/994418194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a881078a0 0x7f0a88107c80 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 -- 172.21.15.89:0/994418194 >> 172.21.15.89:0/994418194 conn(0x7f0a880758c0 msgr2=0x7f0a88075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 -- 172.21.15.89:0/994418194 shutdown_connections 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.314+0000 7f0a8feaf640 1 -- 172.21.15.89:0/994418194 wait complete. 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 Processor -- start 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 -- start start 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a880fff60 0x7f0a88197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:35.317 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a88100920 0x7f0a88197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a881078a0 0x7f0a8819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88110fd0 con 0x7f0a881078a0 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88110e50 con 0x7f0a880fff60 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8feaf640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88111150 con 0x7f0a88100920 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8d44a640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a88100920 0x7f0a88197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a881078a0 0x7f0a8819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a881078a0 0x7f0a8819bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59546/0 (socket says 172.21.15.89:59546) 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8d44a640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a88100920 0x7f0a88197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:41522/0 (socket says 172.21.15.89:41522) 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 -- 172.21.15.89:0/634578306 learned_addr learned my addr 172.21.15.89:0/634578306 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 -- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a88100920 msgr2=0x7f0a88197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8dc4b640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a880fff60 0x7f0a88197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:35.318 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a88100920 0x7f0a88197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 -- 172.21.15.89:0/634578306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a880fff60 msgr2=0x7f0a88197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a880fff60 0x7f0a88197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 -- 172.21.15.89:0/634578306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8819c510 con 0x7f0a881078a0 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8d44a640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a88100920 0x7f0a88197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.315+0000 7f0a8e44c640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a881078a0 0x7f0a8819bde0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000efd0 tx=0x7f0a8000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.316+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003b70 con 0x7f0a881078a0 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.316+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8819c800 con 0x7f0a881078a0 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.316+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0a80003d10 con 0x7f0a881078a0 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.316+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80010840 con 0x7f0a881078a0 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.316+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a881a4090 con 0x7f0a881078a0 2024-03-21T13:28:35.319 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.316+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a880689d0 con 0x7f0a881078a0 2024-03-21T13:28:35.320 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.317+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f0a80004b40 con 0x7f0a881078a0 2024-03-21T13:28:35.320 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.318+0000 7f0a7effd640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0a7406e2b0 0x7f0a74070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:35.320 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.318+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(760..760 src has 1..760) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f0a8008f7b0 con 0x7f0a881078a0 2024-03-21T13:28:35.320 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.318+0000 7f0a8dc4b640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0a7406e2b0 0x7f0a74070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:35.320 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.318+0000 7f0a8dc4b640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0a7406e2b0 0x7f0a74070770 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f0a84036040 tx=0x7f0a84000bc0 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:28:35.322 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.322+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a8005c3f0 con 0x7f0a881078a0 2024-03-21T13:28:35.499 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:35.498+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f0a88198870 con 0x7f0a881078a0 2024-03-21T13:28:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:35 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:35 smithi152 ceph-mon[34051]: osdmap e759: 8 total, 8 up, 8 in 2024-03-21T13:28:35.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:35 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]: dispatch 2024-03-21T13:28:35.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:35 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:35.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:35 smithi089 ceph-mon[30095]: osdmap e759: 8 total, 8 up, 8 in 2024-03-21T13:28:35.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:35 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]: dispatch 2024-03-21T13:28:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:35 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi089-79633-111"}]': finished 2024-03-21T13:28:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:35 smithi089 ceph-mon[39209]: osdmap e759: 8 total, 8 up, 8 in 2024-03-21T13:28:35.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:35 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]: dispatch 2024-03-21T13:28:36.113 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:36.112+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v761) v1 ==== 223+0+0 (secure 0 0 0) 0x7f0a8005fe40 con 0x7f0a881078a0 2024-03-21T13:28:36.229 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:36.229+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f0a88109bc0 con 0x7f0a881078a0 2024-03-21T13:28:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:36 smithi152 ceph-mon[34051]: pgmap v1190: 164 pgs: 164 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]': finished 2024-03-21T13:28:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:36 smithi152 ceph-mon[34051]: osdmap e760: 8 total, 8 up, 8 in 2024-03-21T13:28:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]: dispatch 2024-03-21T13:28:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:36 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:36.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:36 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:28:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[30095]: pgmap v1190: 164 pgs: 164 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:36.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]': finished 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[30095]: osdmap e760: 8 total, 8 up, 8 in 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]: dispatch 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[39209]: pgmap v1190: 164 pgs: 164 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]': finished 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[39209]: osdmap e760: 8 total, 8 up, 8 in 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3199288474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pg_num": 12}]: dispatch 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:28:36.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:36 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:28:37.127 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.126+0000 7f0a7effd640 1 -- 172.21.15.89:0/634578306 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v762) v1 ==== 223+0+0 (secure 0 0 0) 0x7f0a80064670 con 0x7f0a881078a0 2024-03-21T13:28:37.127 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_objects = 10 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.130+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0a7406e2b0 msgr2=0x7f0a74070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.130+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0a7406e2b0 0x7f0a74070770 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f0a84036040 tx=0x7f0a84000bc0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.130+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a881078a0 msgr2=0x7f0a8819bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.130+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a881078a0 0x7f0a8819bde0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000efd0 tx=0x7f0a8000c6a0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 shutdown_connections 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0a7406e2b0 0x7f0a74070770 secure :-1 s=CLOSED pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f0a84036040 tx=0x7f0a84000bc0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a881078a0 0x7f0a8819bde0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.131 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0a88100920 0x7f0a88197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.132 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 --2- 172.21.15.89:0/634578306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a880fff60 0x7f0a88197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.132 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 >> 172.21.15.89:0/634578306 conn(0x7f0a880758c0 msgr2=0x7f0a880fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:28:37.132 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 shutdown_connections 2024-03-21T13:28:37.132 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.131+0000 7f0a8feaf640 1 -- 172.21.15.89:0/634578306 wait complete. 2024-03-21T13:28:37.143 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool application enable f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a rados 2024-03-21T13:28:37.233 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 -- 172.21.15.89:0/3478458870 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 msgr2=0x7fb224106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:37.233 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3478458870 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 0x7fb224106a20 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7fb214009890 tx=0x7fb21402bb70 comp rx=0 tx=0).stop 2024-03-21T13:28:37.233 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 -- 172.21.15.89:0/3478458870 shutdown_connections 2024-03-21T13:28:37.233 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3478458870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb224100f40 0x7fb22410d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.233 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3478458870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 0x7fb224100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.233 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3478458870 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 0x7fb224106a20 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 -- 172.21.15.89:0/3478458870 >> 172.21.15.89:0/3478458870 conn(0x7fb2240776c0 msgr2=0x7fb224077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 -- 172.21.15.89:0/3478458870 shutdown_connections 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 -- 172.21.15.89:0/3478458870 wait complete. 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 Processor -- start 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 -- start start 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.231+0000 7fb22a19a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 0x7fb22410de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb22a19a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb224100f40 0x7fb22410e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb22a19a640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 0x7fb22410e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb22a19a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb224110f50 con 0x7fb224100580 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb22a19a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb224110dd0 con 0x7fb224100f40 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb22a19a640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2241110d0 con 0x7fb224106640 2024-03-21T13:28:37.234 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb2237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 0x7fb22410de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb223fff640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 0x7fb22410e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb2237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 0x7fb22410de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59560/0 (socket says 172.21.15.89:59560) 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb2237fe640 1 -- 172.21.15.89:0/3782397876 learned_addr learned my addr 172.21.15.89:0/3782397876 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb222ffd640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb224100f40 0x7fb22410e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb223fff640 1 -- 172.21.15.89:0/3782397876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb224100f40 msgr2=0x7fb22410e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb223fff640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb224100f40 0x7fb22410e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb223fff640 1 -- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 msgr2=0x7fb22410de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb223fff640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 0x7fb22410de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb223fff640 1 -- 172.21.15.89:0/3782397876 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2241a59b0 con 0x7fb224106640 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb222ffd640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb224100f40 0x7fb22410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb2237fe640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 0x7fb22410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb223fff640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 0x7fb22410e910 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00ca30 tx=0x7fb21c00cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:28:37.235 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c007d90 con 0x7fb224106640 2024-03-21T13:28:37.236 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb21c005020 con 0x7fb224106640 2024-03-21T13:28:37.236 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2241a5ca0 con 0x7fb224106640 2024-03-21T13:28:37.236 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00f6b0 con 0x7fb224106640 2024-03-21T13:28:37.236 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.232+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2241a61f0 con 0x7fb224106640 2024-03-21T13:28:37.236 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.233+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb224068580 con 0x7fb224106640 2024-03-21T13:28:37.237 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.234+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fb21c004a40 con 0x7fb224106640 2024-03-21T13:28:37.237 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.234+0000 7fb220ff9640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fb20c06e2b0 0x7fb20c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:28:37.237 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.234+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(762..762 src has 1..762) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fb21c08f100 con 0x7fb224106640 2024-03-21T13:28:37.237 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.234+0000 7fb2237fe640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fb20c06e2b0 0x7fb20c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:28:37.237 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.235+0000 7fb2237fe640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fb20c06e2b0 0x7fb20c070770 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7fb214005e00 tx=0x7fb21402e040 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:28:37.239 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.239+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb21c014070 con 0x7fb224106640 2024-03-21T13:28:37.416 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:37.416+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"} v 0) v1 -- 0x7fb224108a30 con 0x7fb224106640 2024-03-21T13:28:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:28:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:37 smithi089 ceph-mon[30095]: osdmap e761: 8 total, 8 up, 8 in 2024-03-21T13:28:37.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:37 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:28:37.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:28:37.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:37 smithi089 ceph-mon[39209]: osdmap e761: 8 total, 8 up, 8 in 2024-03-21T13:28:37.496 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:37 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:28:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:28:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:37 smithi152 ceph-mon[34051]: osdmap e761: 8 total, 8 up, 8 in 2024-03-21T13:28:37.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:37 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:28:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:28:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:28:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:28:38.132 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:38.132+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]=0 enabled application 'rados' on pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' v763) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb21c05f710 con 0x7fb224106640 2024-03-21T13:28:38.244 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:38.244+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"} v 0) v1 -- 0x7fb2241084a0 con 0x7fb224106640 2024-03-21T13:28:38.531 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:38 smithi152 ceph-mon[34051]: pgmap v1193: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:38.531 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:28:38.531 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:38 smithi152 ceph-mon[34051]: osdmap e762: 8 total, 8 up, 8 in 2024-03-21T13:28:38.532 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:38 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3782397876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:38.532 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:38 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[30095]: pgmap v1193: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[30095]: osdmap e762: 8 total, 8 up, 8 in 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3782397876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[39209]: pgmap v1193: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/634578306' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[39209]: osdmap e762: 8 total, 8 up, 8 in 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3782397876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:38.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:38 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:38.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:28:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:28:39.147 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.147+0000 7fb220ff9640 1 -- 172.21.15.89:0/3782397876 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]=0 enabled application 'rados' on pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' v764) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb21c063f40 con 0x7fb224106640 2024-03-21T13:28:39.147 INFO:tasks.workunit.client.0.smithi089.stderr:enabled application 'rados' on pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' 2024-03-21T13:28:39.150 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fb20c06e2b0 msgr2=0x7fb20c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:39.150 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fb20c06e2b0 0x7fb20c070770 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7fb214005e00 tx=0x7fb21402e040 comp rx=0 tx=0).stop 2024-03-21T13:28:39.150 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 msgr2=0x7fb22410e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:28:39.150 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 0x7fb22410e910 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00ca30 tx=0x7fb21c00cef0 comp rx=0 tx=0).stop 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 shutdown_connections 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fb20c06e2b0 0x7fb20c070770 unknown :-1 s=CLOSED pgs=3451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb224106640 0x7fb22410e910 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb224100f40 0x7fb22410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 --2- 172.21.15.89:0/3782397876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb224100580 0x7fb22410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 >> 172.21.15.89:0/3782397876 conn(0x7fb2240776c0 msgr2=0x7fb2240ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 shutdown_connections 2024-03-21T13:28:39.151 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:28:39.150+0000 7fb22a19a640 1 -- 172.21.15.89:0/3782397876 wait complete. 2024-03-21T13:28:39.161 INFO:tasks.workunit.client.0.smithi089.stderr:++ seq 1 10 2024-03-21T13:28:39.173 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.173 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj1 /etc/passwd 2024-03-21T13:28:39.213 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.306 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj2 /etc/passwd 2024-03-21T13:28:39.307 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.307 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj3 /etc/passwd 2024-03-21T13:28:39.307 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.307 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj4 /etc/passwd 2024-03-21T13:28:39.320 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.320 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj5 /etc/passwd 2024-03-21T13:28:39.355 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.355 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj6 /etc/passwd 2024-03-21T13:28:39.389 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.389 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj7 /etc/passwd 2024-03-21T13:28:39.423 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.424 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj8 /etc/passwd 2024-03-21T13:28:39.458 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.458 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj9 /etc/passwd 2024-03-21T13:28:39.492 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:28:39.492 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put obj10 /etc/passwd 2024-03-21T13:28:39.526 INFO:tasks.workunit.client.0.smithi089.stderr:+ sleep 30 2024-03-21T13:28:39.570 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]': finished 2024-03-21T13:28:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:39 smithi152 ceph-mon[34051]: osdmap e763: 8 total, 8 up, 8 in 2024-03-21T13:28:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:39 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3782397876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:39 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:39.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]': finished 2024-03-21T13:28:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[30095]: osdmap e763: 8 total, 8 up, 8 in 2024-03-21T13:28:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3782397876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:39.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:39.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]': finished 2024-03-21T13:28:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[39209]: osdmap e763: 8 total, 8 up, 8 in 2024-03-21T13:28:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3782397876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]: dispatch 2024-03-21T13:28:39.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:40.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:40 smithi089 ceph-mon[30095]: pgmap v1196: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:40.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:40 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]': finished 2024-03-21T13:28:40.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:40 smithi089 ceph-mon[30095]: osdmap e764: 8 total, 8 up, 8 in 2024-03-21T13:28:40.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:40 smithi089 ceph-mon[39209]: pgmap v1196: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:40 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]': finished 2024-03-21T13:28:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:40 smithi089 ceph-mon[39209]: osdmap e764: 8 total, 8 up, 8 in 2024-03-21T13:28:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:40 smithi152 ceph-mon[34051]: pgmap v1196: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:28:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:40 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "app": "rados"}]': finished 2024-03-21T13:28:40.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:40 smithi152 ceph-mon[34051]: osdmap e764: 8 total, 8 up, 8 in 2024-03-21T13:28:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:42 smithi152 ceph-mon[34051]: pgmap v1198: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 24 KiB/s wr, 3 op/s 2024-03-21T13:28:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:28:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:28:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:28:42.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:42 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:42.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[30095]: pgmap v1198: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 24 KiB/s wr, 3 op/s 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[39209]: pgmap v1198: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 24 KiB/s wr, 3 op/s 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:28:42.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:42 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:43.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:43.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:43.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:44.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:44 smithi152 ceph-mon[34051]: pgmap v1199: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-21T13:28:44.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:44 smithi089 ceph-mon[30095]: pgmap v1199: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-21T13:28:44.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:44 smithi089 ceph-mon[39209]: pgmap v1199: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-21T13:28:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:46 smithi152 ceph-mon[34051]: pgmap v1200: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 646 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-21T13:28:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:46 smithi152 ceph-mon[34051]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' is full (reached quota's max_objects: 10) 2024-03-21T13:28:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:46 smithi152 ceph-mon[34051]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:28:46.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:46 smithi152 ceph-mon[34051]: osdmap e765: 8 total, 8 up, 8 in 2024-03-21T13:28:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[30095]: pgmap v1200: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 646 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-21T13:28:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[30095]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' is full (reached quota's max_objects: 10) 2024-03-21T13:28:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[30095]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:28:46.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[30095]: osdmap e765: 8 total, 8 up, 8 in 2024-03-21T13:28:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[39209]: pgmap v1200: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 646 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-21T13:28:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[39209]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' is full (reached quota's max_objects: 10) 2024-03-21T13:28:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[39209]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:28:46.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:46 smithi089 ceph-mon[39209]: osdmap e765: 8 total, 8 up, 8 in 2024-03-21T13:28:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:28:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:28:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:28:48.542 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:48 smithi152 ceph-mon[34051]: pgmap v1202: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-21T13:28:48.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:48 smithi089 ceph-mon[30095]: pgmap v1202: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-21T13:28:48.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:48 smithi089 ceph-mon[39209]: pgmap v1202: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-21T13:28:48.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:28:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:28:49.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:49.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:49.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:50.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:50 smithi152 ceph-mon[34051]: pgmap v1203: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-21T13:28:50.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:50 smithi089 ceph-mon[30095]: pgmap v1203: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-21T13:28:50.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:50 smithi089 ceph-mon[39209]: pgmap v1203: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-21T13:28:52.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:52 smithi152 ceph-mon[34051]: pgmap v1204: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:52.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:52 smithi089 ceph-mon[30095]: pgmap v1204: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:52.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:52 smithi089 ceph-mon[39209]: pgmap v1204: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:54.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:54 smithi152 ceph-mon[34051]: pgmap v1205: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:54.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:54 smithi089 ceph-mon[30095]: pgmap v1205: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:54.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:54 smithi089 ceph-mon[39209]: pgmap v1205: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:56.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:56 smithi152 ceph-mon[34051]: pgmap v1206: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:56.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:56 smithi089 ceph-mon[30095]: pgmap v1206: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:56.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:56 smithi089 ceph-mon[39209]: pgmap v1206: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:28:57.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:28:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:28:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:28:58.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:58 smithi152 ceph-mon[34051]: pgmap v1207: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:28:58.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:58.550 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:58 smithi089 ceph-mon[39209]: pgmap v1207: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:28:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:58.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:58 smithi089 ceph-mon[30095]: pgmap v1207: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:28:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:28:58.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:28:58.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:28:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:28:59.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:28:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:59.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:28:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:28:59.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:28:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:00.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:00 smithi152 ceph-mon[34051]: pgmap v1208: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:00.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:00 smithi089 ceph-mon[30095]: pgmap v1208: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:00.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:00 smithi089 ceph-mon[39209]: pgmap v1208: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:02.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:02 smithi152 ceph-mon[34051]: pgmap v1209: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:02.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:02 smithi089 ceph-mon[30095]: pgmap v1209: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:02.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:02 smithi089 ceph-mon[39209]: pgmap v1209: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:04.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:04 smithi152 ceph-mon[34051]: pgmap v1210: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:04.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:04 smithi089 ceph-mon[39209]: pgmap v1210: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:04.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:04 smithi089 ceph-mon[30095]: pgmap v1210: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:06.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:06 smithi152 ceph-mon[34051]: pgmap v1211: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:06.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:06 smithi089 ceph-mon[39209]: pgmap v1211: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:06.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:06 smithi089 ceph-mon[30095]: pgmap v1211: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:07.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:29:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:29:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:29:08.559 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:08 smithi152 ceph-mon[34051]: pgmap v1212: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:08.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:08 smithi089 ceph-mon[39209]: pgmap v1212: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:08.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:08 smithi089 ceph-mon[30095]: pgmap v1212: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:08.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:29:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:29:09.527 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=150611 2024-03-21T13:29:09.528 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a max_objects 100 2024-03-21T13:29:09.528 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put onemore /etc/passwd 2024-03-21T13:29:09.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:09.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:09.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:09.628 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 -- 172.21.15.89:0/1037373177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0101670 msgr2=0x7f8fe0101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:09.628 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/1037373177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0101670 0x7f8fe0101af0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009930 tx=0x7f8fd402bc30 comp rx=0 tx=0).stop 2024-03-21T13:29:09.628 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 -- 172.21.15.89:0/1037373177 shutdown_connections 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/1037373177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0102030 0x7f8fe010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/1037373177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0101670 0x7f8fe0101af0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/1037373177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fe0107680 0x7f8fe0107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 -- 172.21.15.89:0/1037373177 >> 172.21.15.89:0/1037373177 conn(0x7f8fe00fd1a0 msgr2=0x7f8fe00ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 -- 172.21.15.89:0/1037373177 shutdown_connections 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.622+0000 7f8fe86a8640 1 -- 172.21.15.89:0/1037373177 wait complete. 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 Processor -- start 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 -- start start 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0101670 0x7f8fe0197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0102030 0x7f8fe01979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fe0107680 0x7f8fe019bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0110f60 con 0x7f8fe0101670 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0110de0 con 0x7f8fe0107680 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe86a8640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01110e0 con 0x7f8fe0102030 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe6444640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0101670 0x7f8fe0197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe5c43640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0102030 0x7f8fe01979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe6444640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0101670 0x7f8fe0197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53396/0 (socket says 172.21.15.89:53396) 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe6444640 1 -- 172.21.15.89:0/4054373774 learned_addr learned my addr 172.21.15.89:0/4054373774 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe6c45640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fe0107680 0x7f8fe019bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe5c43640 1 -- 172.21.15.89:0/4054373774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fe0107680 msgr2=0x7f8fe019bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe5c43640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fe0107680 0x7f8fe019bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.623+0000 7f8fe5c43640 1 -- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0101670 msgr2=0x7f8fe0197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fe5c43640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0101670 0x7f8fe0197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fe5c43640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe019c450 con 0x7f8fe0102030 2024-03-21T13:29:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fe6444640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0101670 0x7f8fe0197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fe6c45640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fe0107680 0x7f8fe019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fe5c43640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0102030 0x7f8fe01979a0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f8fd402c170 tx=0x7f8fd40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4043020 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8fd4004a50 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe019c6e0 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd40037c0 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.624+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe01a3fd0 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.625+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe0102db0 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.625+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f8fd4046020 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.626+0000 7f8fd37fe640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f8fcc06e2b0 0x7f8fcc070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.626+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(765..765 src has 1..765) v4 ==== 9163+0+0 (secure 0 0 0) 0x7f8fd40b0f20 con 0x7f8fe0102030 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.626+0000 7f8fe6444640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f8fcc06e2b0 0x7f8fcc070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:09.631 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.626+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=766}) v3 -- 0x7f8fcc079f20 con 0x7f8fe0102030 2024-03-21T13:29:09.632 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.626+0000 7f8fe6444640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f8fcc06e2b0 0x7f8fcc070770 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00f3c0 tx=0x7f8fdc00a380 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:09.632 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.631+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8fd404d020 con 0x7f8fe0102030 2024-03-21T13:29:09.808 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:09.807+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f8fe01987b0 con 0x7f8fe0102030 2024-03-21T13:29:10.213 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:10.213+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 7 ==== osd_map(766..766 src has 1..766) v4 ==== 628+0+0 (secure 0 0 0) 0x7f8fd407b470 con 0x7f8fe0102030 2024-03-21T13:29:10.213 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:10.213+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=767}) v3 -- 0x7f8fcc07aa90 con 0x7f8fe0102030 2024-03-21T13:29:10.217 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:10.217+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v766) v1 ==== 225+0+0 (secure 0 0 0) 0x7f8fd407dad0 con 0x7f8fe0102030 2024-03-21T13:29:10.282 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:10.282+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 9 ==== osd_map(767..767 src has 1..767) v4 ==== 628+0+0 (secure 0 0 0) 0x7f8fd40ae760 con 0x7f8fe0102030 2024-03-21T13:29:10.283 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:10.282+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=768}) v3 -- 0x7f8fcc07aec0 con 0x7f8fe0102030 2024-03-21T13:29:10.332 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:10.331+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f8fe0109be0 con 0x7f8fe0102030 2024-03-21T13:29:10.475 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:10 smithi089 ceph-mon[39209]: pgmap v1213: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:10.475 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:10 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4054373774' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:10.476 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:10 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:10.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:10 smithi089 ceph-mon[30095]: pgmap v1213: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:10.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:10 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4054373774' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:10.476 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:10 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:10 smithi152 ceph-mon[34051]: pgmap v1213: 176 pgs: 176 active+clean; 698 KiB data, 868 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:10 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4054373774' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:10.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:10 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:11.289 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.289+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 10 ==== osd_map(768..768 src has 1..768) v4 ==== 628+0+0 (secure 0 0 0) 0x7f8fd407b6d0 con 0x7f8fe0102030 2024-03-21T13:29:11.294 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.294+0000 7f8fd37fe640 1 -- 172.21.15.89:0/4054373774 <== mon.2 v2:172.21.15.89:3301/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v768) v1 ==== 225+0+0 (secure 0 0 0) 0x7f8fd4081520 con 0x7f8fe0102030 2024-03-21T13:29:11.294 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_objects = 100 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f8fcc06e2b0 msgr2=0x7f8fcc070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f8fcc06e2b0 0x7f8fcc070770 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00f3c0 tx=0x7f8fdc00a380 comp rx=0 tx=0).stop 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0102030 msgr2=0x7f8fe01979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0102030 0x7f8fe01979a0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f8fd402c170 tx=0x7f8fd40042f0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 shutdown_connections 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f8fcc06e2b0 0x7f8fcc070770 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fe0107680 0x7f8fe019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe0102030 0x7f8fe01979a0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 --2- 172.21.15.89:0/4054373774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe0101670 0x7f8fe0197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 >> 172.21.15.89:0/4054373774 conn(0x7f8fe00fd1a0 msgr2=0x7f8fe00fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:11.298 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 shutdown_connections 2024-03-21T13:29:11.299 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.297+0000 7f8fe86a8640 1 -- 172.21.15.89:0/4054373774 wait complete. 2024-03-21T13:29:11.308 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 150611 2024-03-21T13:29:11.309 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 0 -ne 0 ']' 2024-03-21T13:29:11.309 INFO:tasks.workunit.client.0.smithi089.stderr:+ true 2024-03-21T13:29:11.309 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put twomore /etc/passwd 2024-03-21T13:29:11.346 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a max_bytes 100 2024-03-21T13:29:11.433 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 -- 172.21.15.89:0/2276538478 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 msgr2=0x7fe52810d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:11.433 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 --2- 172.21.15.89:0/2276538478 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 0x7fe52810d400 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7fe520009930 tx=0x7fe52002bc30 comp rx=0 tx=0).stop 2024-03-21T13:29:11.433 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 -- 172.21.15.89:0/2276538478 shutdown_connections 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 --2- 172.21.15.89:0/2276538478 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 0x7fe52810d400 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 --2- 172.21.15.89:0/2276538478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe528100580 0x7fe528100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 --2- 172.21.15.89:0/2276538478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528106640 0x7fe528106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 -- 172.21.15.89:0/2276538478 >> 172.21.15.89:0/2276538478 conn(0x7fe5280776c0 msgr2=0x7fe528077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 -- 172.21.15.89:0/2276538478 shutdown_connections 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.430+0000 7fe52f102640 1 -- 172.21.15.89:0/2276538478 wait complete. 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 Processor -- start 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 -- start start 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528100580 0x7fe5281974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 0x7fe5281979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe528106640 0x7fe52819bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528110f10 con 0x7fe528100580 2024-03-21T13:29:11.434 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528110d90 con 0x7fe528106640 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52f102640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe528111090 con 0x7fe528100f40 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52ce9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528100580 0x7fe5281974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe527fff640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 0x7fe5281979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52ce9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528100580 0x7fe5281974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53424/0 (socket says 172.21.15.89:53424) 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52ce9e640 1 -- 172.21.15.89:0/1126514531 learned_addr learned my addr 172.21.15.89:0/1126514531 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.431+0000 7fe52d69f640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe528106640 0x7fe52819bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52ce9e640 1 -- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 msgr2=0x7fe5281979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52ce9e640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 0x7fe5281979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52ce9e640 1 -- 172.21.15.89:0/1126514531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe528106640 msgr2=0x7fe52819bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52ce9e640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe528106640 0x7fe52819bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52ce9e640 1 -- 172.21.15.89:0/1126514531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52819c4a0 con 0x7fe528100580 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52d69f640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe528106640 0x7fe52819bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T13:29:11.435 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe527fff640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 0x7fe5281979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52ce9e640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528100580 0x7fe5281974b0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fe518009d50 tx=0x7fe51800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518003b70 con 0x7fe528100580 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe518003d10 con 0x7fe528100580 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52819c790 con 0x7fe528100580 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180107d0 con 0x7fe528100580 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.432+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5281a4020 con 0x7fe528100580 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.433+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528068580 con 0x7fe528100580 2024-03-21T13:29:11.436 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.433+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fe518004000 con 0x7fe528100580 2024-03-21T13:29:11.437 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.434+0000 7fe525ffb640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fe51406e2b0 0x7fe514070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:11.437 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.434+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(768..768 src has 1..768) v4 ==== 9163+0+0 (secure 0 0 0) 0x7fe51808f5e0 con 0x7fe528100580 2024-03-21T13:29:11.437 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.434+0000 7fe527fff640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fe51406e2b0 0x7fe514070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:11.437 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.434+0000 7fe527fff640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fe51406e2b0 0x7fe514070770 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fe528077de0 tx=0x7fe51000b390 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:11.438 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.438+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe51805c190 con 0x7fe528100580 2024-03-21T13:29:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]': finished 2024-03-21T13:29:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:11 smithi152 ceph-mon[34051]: osdmap e766: 8 total, 8 up, 8 in 2024-03-21T13:29:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:11 smithi152 ceph-mon[34051]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' no longer out of quota; removing NO_QUOTA flag 2024-03-21T13:29:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:11 smithi152 ceph-mon[34051]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:29:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:11 smithi152 ceph-mon[34051]: osdmap e767: 8 total, 8 up, 8 in 2024-03-21T13:29:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:11 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4054373774' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:11.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:11 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:11.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]': finished 2024-03-21T13:29:11.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[30095]: osdmap e766: 8 total, 8 up, 8 in 2024-03-21T13:29:11.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[30095]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' no longer out of quota; removing NO_QUOTA flag 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[30095]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[30095]: osdmap e767: 8 total, 8 up, 8 in 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4054373774' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]': finished 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[39209]: osdmap e766: 8 total, 8 up, 8 in 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[39209]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' no longer out of quota; removing NO_QUOTA flag 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[39209]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[39209]: osdmap e767: 8 total, 8 up, 8 in 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4054373774' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:11.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:11 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]: dispatch 2024-03-21T13:29:11.617 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:11.617+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fe528108a50 con 0x7fe528100580 2024-03-21T13:29:12.290 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:12.290+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v769) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe51805fbe0 con 0x7fe528100580 2024-03-21T13:29:12.403 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:12.403+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fe5281084c0 con 0x7fe528100580 2024-03-21T13:29:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:12 smithi152 ceph-mon[34051]: pgmap v1216: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:29:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:12 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]': finished 2024-03-21T13:29:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:12 smithi152 ceph-mon[34051]: osdmap e768: 8 total, 8 up, 8 in 2024-03-21T13:29:12.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:12 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-21T13:29:12.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[39209]: pgmap v1216: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:29:12.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]': finished 2024-03-21T13:29:12.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[39209]: osdmap e768: 8 total, 8 up, 8 in 2024-03-21T13:29:12.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-21T13:29:12.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[30095]: pgmap v1216: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:29:12.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "100"}]': finished 2024-03-21T13:29:12.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[30095]: osdmap e768: 8 total, 8 up, 8 in 2024-03-21T13:29:12.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:12 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-21T13:29:13.303 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.303+0000 7fe525ffb640 1 -- 172.21.15.89:0/1126514531 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v770) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe518064410 con 0x7fe528100580 2024-03-21T13:29:13.304 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_bytes = 100 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:29:13.307 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fe51406e2b0 msgr2=0x7fe514070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:13.307 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fe51406e2b0 0x7fe514070770 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fe528077de0 tx=0x7fe51000b390 comp rx=0 tx=0).stop 2024-03-21T13:29:13.307 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528100580 msgr2=0x7fe5281974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528100580 0x7fe5281974b0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fe518009d50 tx=0x7fe51800c6a0 comp rx=0 tx=0).stop 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 shutdown_connections 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fe51406e2b0 0x7fe514070770 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe528106640 0x7fe52819bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe528100f40 0x7fe5281979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 --2- 172.21.15.89:0/1126514531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe528100580 0x7fe5281974b0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 >> 172.21.15.89:0/1126514531 conn(0x7fe5280776c0 msgr2=0x7fe5280ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 shutdown_connections 2024-03-21T13:29:13.308 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:13.307+0000 7fe52f102640 1 -- 172.21.15.89:0/1126514531 wait complete. 2024-03-21T13:29:13.319 INFO:tasks.workunit.client.0.smithi089.stderr:+ sleep 30 2024-03-21T13:29:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]': finished 2024-03-21T13:29:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:13 smithi152 ceph-mon[34051]: osdmap e769: 8 total, 8 up, 8 in 2024-03-21T13:29:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:13 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-21T13:29:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:13 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:13.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:13.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]': finished 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[39209]: osdmap e769: 8 total, 8 up, 8 in 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]': finished 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[30095]: osdmap e769: 8 total, 8 up, 8 in 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:13.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:14 smithi152 ceph-mon[34051]: pgmap v1219: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:14 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]': finished 2024-03-21T13:29:14.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:14 smithi152 ceph-mon[34051]: osdmap e770: 8 total, 8 up, 8 in 2024-03-21T13:29:14.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:14 smithi089 ceph-mon[39209]: pgmap v1219: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:14.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:14 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]': finished 2024-03-21T13:29:14.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:14 smithi089 ceph-mon[39209]: osdmap e770: 8 total, 8 up, 8 in 2024-03-21T13:29:14.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:14 smithi089 ceph-mon[30095]: pgmap v1219: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:14.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:14 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/1126514531' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "100"}]': finished 2024-03-21T13:29:14.576 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:14 smithi089 ceph-mon[30095]: osdmap e770: 8 total, 8 up, 8 in 2024-03-21T13:29:15.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:15 smithi152 ceph-mon[34051]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' is full (reached quota's max_bytes: 100 B) 2024-03-21T13:29:15.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:15 smithi152 ceph-mon[34051]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:29:15.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:15 smithi152 ceph-mon[34051]: osdmap e771: 8 total, 8 up, 8 in 2024-03-21T13:29:15.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:15 smithi089 ceph-mon[39209]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' is full (reached quota's max_bytes: 100 B) 2024-03-21T13:29:15.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:15 smithi089 ceph-mon[39209]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:29:15.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:15 smithi089 ceph-mon[39209]: osdmap e771: 8 total, 8 up, 8 in 2024-03-21T13:29:15.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:15 smithi089 ceph-mon[30095]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' is full (reached quota's max_bytes: 100 B) 2024-03-21T13:29:15.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:15 smithi089 ceph-mon[30095]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:29:15.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:15 smithi089 ceph-mon[30095]: osdmap e771: 8 total, 8 up, 8 in 2024-03-21T13:29:16.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:16 smithi152 ceph-mon[34051]: pgmap v1221: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-03-21T13:29:16.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:16 smithi089 ceph-mon[30095]: pgmap v1221: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-03-21T13:29:16.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:16 smithi089 ceph-mon[39209]: pgmap v1221: 176 pgs: 176 active+clean; 709 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-03-21T13:29:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:29:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:29:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:29:18.571 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:18 smithi152 ceph-mon[34051]: pgmap v1223: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 887 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-21T13:29:18.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:29:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:29:18.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:18 smithi089 ceph-mon[30095]: pgmap v1223: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 887 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-21T13:29:18.574 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:18 smithi089 ceph-mon[39209]: pgmap v1223: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 887 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-21T13:29:19.575 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:19.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:19.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:20.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:20 smithi152 ceph-mon[34051]: pgmap v1224: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 756 B/s rd, 1.8 KiB/s wr, 0 op/s 2024-03-21T13:29:20.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:20 smithi089 ceph-mon[39209]: pgmap v1224: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 756 B/s rd, 1.8 KiB/s wr, 0 op/s 2024-03-21T13:29:20.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:20 smithi089 ceph-mon[30095]: pgmap v1224: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 756 B/s rd, 1.8 KiB/s wr, 0 op/s 2024-03-21T13:29:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:22 smithi152 ceph-mon[34051]: pgmap v1225: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:29:22.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:22 smithi089 ceph-mon[39209]: pgmap v1225: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:29:22.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:22 smithi089 ceph-mon[30095]: pgmap v1225: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:29:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:24 smithi152 ceph-mon[34051]: pgmap v1226: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-21T13:29:24.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:24 smithi089 ceph-mon[30095]: pgmap v1226: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-21T13:29:24.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:24 smithi089 ceph-mon[39209]: pgmap v1226: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-21T13:29:26.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:26 smithi152 ceph-mon[34051]: pgmap v1227: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-21T13:29:26.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:26 smithi089 ceph-mon[30095]: pgmap v1227: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-21T13:29:26.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:26 smithi089 ceph-mon[39209]: pgmap v1227: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-21T13:29:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:29:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:29:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:29:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:28 smithi152 ceph-mon[34051]: pgmap v1228: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-21T13:29:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:28 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:28.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:29:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:29:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:28 smithi089 ceph-mon[30095]: pgmap v1228: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-21T13:29:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:28 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:28 smithi089 ceph-mon[39209]: pgmap v1228: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-21T13:29:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:28 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:29.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:29.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:30 smithi152 ceph-mon[34051]: pgmap v1229: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:30 smithi089 ceph-mon[30095]: pgmap v1229: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:30.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:30 smithi089 ceph-mon[39209]: pgmap v1229: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:32 smithi152 ceph-mon[34051]: pgmap v1230: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:32.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:32 smithi089 ceph-mon[39209]: pgmap v1230: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:32.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:32 smithi089 ceph-mon[30095]: pgmap v1230: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:34 smithi152 ceph-mon[34051]: pgmap v1231: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:34.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:34 smithi089 ceph-mon[30095]: pgmap v1231: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:34.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:34 smithi089 ceph-mon[39209]: pgmap v1231: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:36.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:36 smithi152 ceph-mon[34051]: pgmap v1232: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:36.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:36 smithi089 ceph-mon[39209]: pgmap v1232: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:36.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:36 smithi089 ceph-mon[30095]: pgmap v1232: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:37.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:29:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:29:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:29:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:38 smithi152 ceph-mon[34051]: pgmap v1233: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:38.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:29:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:29:38.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:38 smithi089 ceph-mon[30095]: pgmap v1233: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:38.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:38 smithi089 ceph-mon[39209]: pgmap v1233: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:39.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:39.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:40.726 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:40 smithi089 ceph-mon[30095]: pgmap v1234: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:40.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:40 smithi089 ceph-mon[39209]: pgmap v1234: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:40 smithi152 ceph-mon[34051]: pgmap v1234: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:42 smithi152 ceph-mon[34051]: pgmap v1235: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:42 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:29:42.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:42 smithi089 ceph-mon[39209]: pgmap v1235: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:42.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:42 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:29:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:42 smithi089 ceph-mon[30095]: pgmap v1235: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:29:42.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:42 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:29:43.321 INFO:tasks.workunit.client.0.smithi089.stderr:+ pid=150828 2024-03-21T13:29:43.321 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a max_bytes 0 2024-03-21T13:29:43.321 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put two /etc/passwd 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 -- 172.21.15.89:0/518665359 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c10bce0 msgr2=0x7fbd0c10e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 --2- 172.21.15.89:0/518665359 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c10bce0 0x7fbd0c10e110 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009be0 tx=0x7fbd0402bf80 comp rx=0 tx=0).stop 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 -- 172.21.15.89:0/518665359 shutdown_connections 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 --2- 172.21.15.89:0/518665359 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c10bce0 0x7fbd0c10e110 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 --2- 172.21.15.89:0/518665359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd0c103230 0x7fbd0c10b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 --2- 172.21.15.89:0/518665359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd0c102910 0x7fbd0c102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 -- 172.21.15.89:0/518665359 >> 172.21.15.89:0/518665359 conn(0x7fbd0c077780 msgr2=0x7fbd0c077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:43.413 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 -- 172.21.15.89:0/518665359 shutdown_connections 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 -- 172.21.15.89:0/518665359 wait complete. 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 Processor -- start 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.410+0000 7fbd11b32640 1 -- start start 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c102910 0x7fbd0c100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd0c103230 0x7fbd0c100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd0c10bce0 0x7fbd0c100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c110f40 con 0x7fbd0c10bce0 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c110dc0 con 0x7fbd0c103230 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c1110c0 con 0x7fbd0c102910 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0b7fe640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c102910 0x7fbd0c100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd0c10bce0 0x7fbd0c100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:43.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0b7fe640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c102910 0x7fbd0c100560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:59646/0 (socket says 172.21.15.89:59646) 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0b7fe640 1 -- 172.21.15.89:0/4129726467 learned_addr learned my addr 172.21.15.89:0/4129726467 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd02ffd640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd0c103230 0x7fbd0c100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0bfff640 1 -- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c102910 msgr2=0x7fbd0c100560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0bfff640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c102910 0x7fbd0c100560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0bfff640 1 -- 172.21.15.89:0/4129726467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd0c103230 msgr2=0x7fbd0c100aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0bfff640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd0c103230 0x7fbd0c100aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0bfff640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c1a59a0 con 0x7fbd0c10bce0 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0b7fe640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c102910 0x7fbd0c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd02ffd640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd0c103230 0x7fbd0c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd0bfff640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd0c10bce0 0x7fbd0c100fe0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fbd0402c4c0 tx=0x7fbd04003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd04043020 con 0x7fbd0c10bce0 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbd04004040 con 0x7fbd0c10bce0 2024-03-21T13:29:43.415 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c1a5c30 con 0x7fbd0c10bce0 2024-03-21T13:29:43.416 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd04034690 con 0x7fbd0c10bce0 2024-03-21T13:29:43.416 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.411+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c1a6180 con 0x7fbd0c10bce0 2024-03-21T13:29:43.416 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.412+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c0689d0 con 0x7fbd0c10bce0 2024-03-21T13:29:43.416 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.413+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fbd04046020 con 0x7fbd0c10bce0 2024-03-21T13:29:43.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.413+0000 7fbd097fa640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fbcf806e2b0 0x7fbcf8070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:43.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.413+0000 7fbd0b7fe640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fbcf806e2b0 0x7fbcf8070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:43.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.414+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(771..771 src has 1..771) v4 ==== 9163+0+0 (secure 0 0 0) 0x7fbd040b1650 con 0x7fbd0c10bce0 2024-03-21T13:29:43.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.414+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=772}) v3 -- 0x7fbcf8079f20 con 0x7fbd0c10bce0 2024-03-21T13:29:43.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.414+0000 7fbd0b7fe640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fbcf806e2b0 0x7fbcf8070770 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0096c0 tx=0x7fbcfc009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:43.418 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.418+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd0404d020 con 0x7fbd0c10bce0 2024-03-21T13:29:43.598 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:43.598+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fbd0c1064e0 con 0x7fbd0c10bce0 2024-03-21T13:29:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:29:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:29:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:43 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:43.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:29:43.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:29:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:29:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:29:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:44.388 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:44.388+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v772) v1 ==== 217+0+0 (secure 0 0 0) 0x7fbd0407e200 con 0x7fbd0c10bce0 2024-03-21T13:29:44.391 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:44.391+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 8 ==== osd_map(772..772 src has 1..772) v4 ==== 628+0+0 (secure 0 0 0) 0x7fbd040e5740 con 0x7fbd0c10bce0 2024-03-21T13:29:44.391 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:44.391+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=773}) v3 -- 0x7fbcf807ae80 con 0x7fbd0c10bce0 2024-03-21T13:29:44.503 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:44.503+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fbd0c1060f0 con 0x7fbd0c10bce0 2024-03-21T13:29:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:44 smithi152 ceph-mon[34051]: pgmap v1236: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:44 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:29:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:44 smithi089 ceph-mon[30095]: pgmap v1236: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:44 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:29:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:44 smithi089 ceph-mon[39209]: pgmap v1236: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:29:44.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:44 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:29:45.289 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.288+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v773) v1 ==== 217+0+0 (secure 0 0 0) 0x7fbd04081c50 con 0x7fbd0c10bce0 2024-03-21T13:29:45.289 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_bytes = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:29:45.291 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.291+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 <== mon.0 v2:172.21.15.89:3300/0 10 ==== osd_map(773..773 src has 1..773) v4 ==== 628+0+0 (secure 0 0 0) 0x7fbd0407bba0 con 0x7fbd0c10bce0 2024-03-21T13:29:45.291 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.291+0000 7fbd097fa640 1 -- 172.21.15.89:0/4129726467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=774}) v3 -- 0x7fbcf807af30 con 0x7fbd0c10bce0 2024-03-21T13:29:45.292 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fbcf806e2b0 msgr2=0x7fbcf8070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:45.292 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fbcf806e2b0 0x7fbcf8070770 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0096c0 tx=0x7fbcfc009260 comp rx=0 tx=0).stop 2024-03-21T13:29:45.292 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd0c10bce0 msgr2=0x7fbd0c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:45.292 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd0c10bce0 0x7fbd0c100fe0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fbd0402c4c0 tx=0x7fbd04003ce0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.292 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 shutdown_connections 2024-03-21T13:29:45.292 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fbcf806e2b0 0x7fbcf8070770 unknown :-1 s=CLOSED pgs=3467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.292 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd0c10bce0 0x7fbd0c100fe0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.293 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd0c103230 0x7fbd0c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.293 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 --2- 172.21.15.89:0/4129726467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd0c102910 0x7fbd0c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.293 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 >> 172.21.15.89:0/4129726467 conn(0x7fbd0c077780 msgr2=0x7fbd0c1036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:45.293 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 shutdown_connections 2024-03-21T13:29:45.293 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.292+0000 7fbd11b32640 1 -- 172.21.15.89:0/4129726467 wait complete. 2024-03-21T13:29:45.308 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a max_objects 0 2024-03-21T13:29:45.397 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 -- 172.21.15.89:0/1390165029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb5010bce0 msgr2=0x7fcb5010e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:45.397 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 --2- 172.21.15.89:0/1390165029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb5010bce0 0x7fcb5010e110 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009ca0 tx=0x7fcb4802c180 comp rx=0 tx=0).stop 2024-03-21T13:29:45.397 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 -- 172.21.15.89:0/1390165029 shutdown_connections 2024-03-21T13:29:45.397 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 --2- 172.21.15.89:0/1390165029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb5010bce0 0x7fcb5010e110 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.397 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 --2- 172.21.15.89:0/1390165029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb50103230 0x7fcb5010b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.397 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 --2- 172.21.15.89:0/1390165029 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb50102910 0x7fcb50102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.397 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 -- 172.21.15.89:0/1390165029 >> 172.21.15.89:0/1390165029 conn(0x7fcb50077690 msgr2=0x7fcb50077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 -- 172.21.15.89:0/1390165029 shutdown_connections 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.395+0000 7fcb57585640 1 -- 172.21.15.89:0/1390165029 wait complete. 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 Processor -- start 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 -- start start 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb50102910 0x7fcb50197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb50103230 0x7fcb50197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb5010bce0 0x7fcb5019bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50110f40 con 0x7fcb50102910 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50110dc0 con 0x7fcb50103230 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb57585640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501110c0 con 0x7fcb5010bce0 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb55b22640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb5010bce0 0x7fcb5019bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:45.398 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb55b22640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb5010bce0 0x7fcb5019bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:59662/0 (socket says 172.21.15.89:59662) 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb55b22640 1 -- 172.21.15.89:0/805299379 learned_addr learned my addr 172.21.15.89:0/805299379 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb54b20640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb50103230 0x7fcb50197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.396+0000 7fcb55321640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb50102910 0x7fcb50197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.397+0000 7fcb55b22640 1 -- 172.21.15.89:0/805299379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb50103230 msgr2=0x7fcb50197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.397+0000 7fcb55b22640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb50103230 0x7fcb50197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.397+0000 7fcb55b22640 1 -- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb50102910 msgr2=0x7fcb50197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.397+0000 7fcb55b22640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb50102910 0x7fcb50197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.397+0000 7fcb55b22640 1 -- 172.21.15.89:0/805299379 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5019c430 con 0x7fcb5010bce0 2024-03-21T13:29:45.399 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.397+0000 7fcb55b22640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb5010bce0 0x7fcb5019bd00 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fcb48004af0 tx=0x7fcb48035df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.399+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48039040 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.399+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcb48044070 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.399+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5019c6c0 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.399+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48032810 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.399+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb501a3fb0 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.400+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb50067cf0 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.400+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fcb48003b70 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.401+0000 7fcb467fc640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcb3c06e2b0 0x7fcb3c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.401+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(773..773 src has 1..773) v4 ==== 9163+0+0 (secure 0 0 0) 0x7fcb480b13a0 con 0x7fcb5010bce0 2024-03-21T13:29:45.401 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.401+0000 7fcb55321640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcb3c06e2b0 0x7fcb3c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:45.402 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.401+0000 7fcb55321640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcb3c06e2b0 0x7fcb3c070770 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009b60 tx=0x7fcb4c005f90 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:45.406 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.405+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb4807dfd0 con 0x7fcb5010bce0 2024-03-21T13:29:45.581 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:45.581+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fcb50106dd0 con 0x7fcb5010bce0 2024-03-21T13:29:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:29:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:45 smithi152 ceph-mon[34051]: osdmap e772: 8 total, 8 up, 8 in 2024-03-21T13:29:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:29:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:45 smithi152 ceph-mon[34051]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' no longer out of quota; removing NO_QUOTA flag 2024-03-21T13:29:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:45 smithi152 ceph-mon[34051]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:29:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:45 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:29:45.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:45 smithi152 ceph-mon[34051]: osdmap e773: 8 total, 8 up, 8 in 2024-03-21T13:29:45.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:29:45.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[30095]: osdmap e772: 8 total, 8 up, 8 in 2024-03-21T13:29:45.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[30095]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' no longer out of quota; removing NO_QUOTA flag 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[30095]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[30095]: osdmap e773: 8 total, 8 up, 8 in 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[39209]: osdmap e772: 8 total, 8 up, 8 in 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[39209]: pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' no longer out of quota; removing NO_QUOTA flag 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[39209]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4129726467' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:29:45.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:45 smithi089 ceph-mon[39209]: osdmap e773: 8 total, 8 up, 8 in 2024-03-21T13:29:46.408 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:46.407+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v774) v1 ==== 221+0+0 (secure 0 0 0) 0x7fcb48081a20 con 0x7fcb5010bce0 2024-03-21T13:29:46.517 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:46.517+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fcb50106840 con 0x7fcb5010bce0 2024-03-21T13:29:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:46 smithi152 ceph-mon[34051]: pgmap v1238: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:29:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:46 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/805299379' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:46 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:46 smithi089 ceph-mon[30095]: pgmap v1238: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:29:46.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:46 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/805299379' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:46 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:46 smithi089 ceph-mon[39209]: pgmap v1238: 176 pgs: 176 active+clean; 721 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:29:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:46 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/805299379' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:46.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:46 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:47.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.413+0000 7fcb467fc640 1 -- 172.21.15.89:0/805299379 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v775) v1 ==== 221+0+0 (secure 0 0 0) 0x7fcb48086250 con 0x7fcb5010bce0 2024-03-21T13:29:47.414 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_objects = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:29:47.416 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcb3c06e2b0 msgr2=0x7fcb3c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcb3c06e2b0 0x7fcb3c070770 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009b60 tx=0x7fcb4c005f90 comp rx=0 tx=0).stop 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb5010bce0 msgr2=0x7fcb5019bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb5010bce0 0x7fcb5019bd00 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fcb48004af0 tx=0x7fcb48035df0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 shutdown_connections 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fcb3c06e2b0 0x7fcb3c070770 unknown :-1 s=CLOSED pgs=3468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb5010bce0 0x7fcb5019bd00 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb50103230 0x7fcb50197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 --2- 172.21.15.89:0/805299379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb50102910 0x7fcb50197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 >> 172.21.15.89:0/805299379 conn(0x7fcb50077690 msgr2=0x7fcb500fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 shutdown_connections 2024-03-21T13:29:47.417 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.416+0000 7fcb57585640 1 -- 172.21.15.89:0/805299379 wait complete. 2024-03-21T13:29:47.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ wait 150828 2024-03-21T13:29:47.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 0 -ne 0 ']' 2024-03-21T13:29:47.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ true 2024-03-21T13:29:47.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put three /etc/passwd 2024-03-21T13:29:47.465 INFO:tasks.workunit.client.0.smithi089.stderr:++ uuidgen 2024-03-21T13:29:47.466 INFO:tasks.workunit.client.0.smithi089.stderr:+ pp=cf6754ed-1d53-435e-bb0e-835c0a37d941 2024-03-21T13:29:47.466 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool create cf6754ed-1d53-435e-bb0e-835c0a37d941 12 2024-03-21T13:29:47.555 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2031154752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 msgr2=0x7f9eec102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:47.555 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2031154752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec102cf0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009900 tx=0x7f9ee802bd60 comp rx=0 tx=0).stop 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2031154752 shutdown_connections 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2031154752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec10bc50 0x7f9eec10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2031154752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9eec103230 0x7f9eec10b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2031154752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec102cf0 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2031154752 >> 172.21.15.89:0/2031154752 conn(0x7f9eec0776c0 msgr2=0x7f9eec077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2031154752 shutdown_connections 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.552+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2031154752 wait complete. 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 Processor -- start 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 -- start start 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:47.556 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9eec103230 0x7f9eec1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec10bc50 0x7f9eec19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec110f20 con 0x7f9eec10bc50 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec110da0 con 0x7f9eec103230 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef45bc640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec1110a0 con 0x7f9eec102910 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef2358640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef2b59640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec10bc50 0x7f9eec19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef2358640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:59728/0 (socket says 172.21.15.89:59728) 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef2358640 1 -- 172.21.15.89:0/2747277724 learned_addr learned my addr 172.21.15.89:0/2747277724 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.553+0000 7f9ef1b57640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9eec103230 0x7f9eec1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef2358640 1 -- 172.21.15.89:0/2747277724 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9eec103230 msgr2=0x7f9eec1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef2358640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9eec103230 0x7f9eec1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef2358640 1 -- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec10bc50 msgr2=0x7f9eec19bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef2358640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec10bc50 0x7f9eec19bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef2358640 1 -- 172.21.15.89:0/2747277724 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec19c450 con 0x7f9eec102910 2024-03-21T13:29:47.557 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef2b59640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec10bc50 0x7f9eec19bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef2358640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec197460 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8000c00 tx=0x7f9ee80045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8043020 con 0x7f9eec102910 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec19c6e0 con 0x7f9eec102910 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9ee802dbe0 con 0x7f9eec102910 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8003850 con 0x7f9eec102910 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.554+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec1a3fd0 con 0x7f9eec102910 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.555+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec068580 con 0x7f9eec102910 2024-03-21T13:29:47.558 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.555+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f9ee8046020 con 0x7f9eec102910 2024-03-21T13:29:47.559 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.556+0000 7f9ee37fe640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f9ed806e2b0 0x7f9ed8070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:47.559 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.556+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(775..775 src has 1..775) v4 ==== 9163+0+0 (secure 0 0 0) 0x7f9ee80b1000 con 0x7f9eec102910 2024-03-21T13:29:47.559 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.556+0000 7f9ef1b57640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f9ed806e2b0 0x7f9ed8070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:47.559 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.556+0000 7f9ef1b57640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f9ed806e2b0 0x7f9ed8070770 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f9edc005e10 tx=0x7f9edc005da0 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:47.560 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.560+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ee804d020 con 0x7f9eec102910 2024-03-21T13:29:47.736 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:47.736+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12} v 0) v1 -- 0x7f9eec106dd0 con 0x7f9eec102910 2024-03-21T13:29:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:29:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:47 smithi152 ceph-mon[34051]: osdmap e774: 8 total, 8 up, 8 in 2024-03-21T13:29:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:47 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/805299379' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:47.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:47 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:29:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:29:47.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:29:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[30095]: osdmap e774: 8 total, 8 up, 8 in 2024-03-21T13:29:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/805299379' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:47.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:29:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[39209]: osdmap e774: 8 total, 8 up, 8 in 2024-03-21T13:29:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/805299379' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:47.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:47 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:29:48.414 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.414+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]=0 pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' created v776) v1 ==== 176+0+0 (secure 0 0 0) 0x7f9ee807dbb0 con 0x7f9eec102910 2024-03-21T13:29:48.522 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.522+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12} v 0) v1 -- 0x7f9eec106960 con 0x7f9eec102910 2024-03-21T13:29:48.523 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.523+0000 7f9ee37fe640 1 -- 172.21.15.89:0/2747277724 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]=0 pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' already exists v776) v1 ==== 183+0+0 (secure 0 0 0) 0x7f9ee8081600 con 0x7f9eec102910 2024-03-21T13:29:48.523 INFO:tasks.workunit.client.0.smithi089.stderr:pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' already exists 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f9ed806e2b0 msgr2=0x7f9ed8070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f9ed806e2b0 0x7f9ed8070770 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f9edc005e10 tx=0x7f9edc005da0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 msgr2=0x7f9eec197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec197460 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8000c00 tx=0x7f9ee80045e0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 shutdown_connections 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f9ed806e2b0 0x7f9ed8070770 unknown :-1 s=CLOSED pgs=3470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec10bc50 0x7f9eec19bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9eec103230 0x7f9eec1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.526 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 --2- 172.21.15.89:0/2747277724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9eec102910 0x7f9eec197460 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.527 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 >> 172.21.15.89:0/2747277724 conn(0x7f9eec0776c0 msgr2=0x7f9eec0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:48.527 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 shutdown_connections 2024-03-21T13:29:48.527 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.525+0000 7f9ef45bc640 1 -- 172.21.15.89:0/2747277724 wait complete. 2024-03-21T13:29:48.534 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool application enable cf6754ed-1d53-435e-bb0e-835c0a37d941 rados 2024-03-21T13:29:48.617 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 -- 172.21.15.89:0/2323102899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98100590 msgr2=0x7fde98100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:48.617 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 --2- 172.21.15.89:0/2323102899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98100590 0x7fde98100a10 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009900 tx=0x7fde8c02bc30 comp rx=0 tx=0).stop 2024-03-21T13:29:48.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 -- 172.21.15.89:0/2323102899 shutdown_connections 2024-03-21T13:29:48.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 --2- 172.21.15.89:0/2323102899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100f50 0x7fde9810d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 --2- 172.21.15.89:0/2323102899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98100590 0x7fde98100a10 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 --2- 172.21.15.89:0/2323102899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde98106650 0x7fde98106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 -- 172.21.15.89:0/2323102899 >> 172.21.15.89:0/2323102899 conn(0x7fde98077720 msgr2=0x7fde98077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:48.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 -- 172.21.15.89:0/2323102899 shutdown_connections 2024-03-21T13:29:48.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.617+0000 7fde9ce15640 1 -- 172.21.15.89:0/2323102899 wait complete. 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 Processor -- start 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 -- start start 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100590 0x7fde98196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde98100f50 0x7fde981965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98106650 0x7fde98196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9810fe60 con 0x7fde98106650 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9810fce0 con 0x7fde98100590 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9ce15640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9810ffe0 con 0x7fde98100f50 2024-03-21T13:29:48.621 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98106650 0x7fde98196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde95d9b640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde98100f50 0x7fde981965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98106650 0x7fde98196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56654/0 (socket says 172.21.15.89:56654) 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 -- 172.21.15.89:0/4107355026 learned_addr learned my addr 172.21.15.89:0/4107355026 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde9659c640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100590 0x7fde98196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 -- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde98100f50 msgr2=0x7fde981965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde98100f50 0x7fde981965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 -- 172.21.15.89:0/4107355026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100590 msgr2=0x7fde98196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100590 0x7fde98196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde96d9d640 1 -- 172.21.15.89:0/4107355026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde981a5900 con 0x7fde98106650 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.618+0000 7fde95d9b640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde98100f50 0x7fde981965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.619+0000 7fde96d9d640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98106650 0x7fde98196b10 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fde9000eaa0 tx=0x7fde9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.619+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9000cdb0 con 0x7fde98106650 2024-03-21T13:29:48.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.619+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde981a5bf0 con 0x7fde98106650 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.619+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fde90005020 con 0x7fde98106650 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.619+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde900116d0 con 0x7fde98106650 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.619+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde981a60c0 con 0x7fde98106650 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.620+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde980689d0 con 0x7fde98106650 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.620+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7fde90004a40 con 0x7fde98106650 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.621+0000 7fde877fe640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fde8006e2b0 0x7fde80070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.621+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(776..776 src has 1..776) v4 ==== 9538+0+0 (secure 0 0 0) 0x7fde9008f280 con 0x7fde98106650 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.621+0000 7fde9659c640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fde8006e2b0 0x7fde80070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:48.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.621+0000 7fde9659c640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fde8006e2b0 0x7fde80070770 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fde88009bd0 tx=0x7fde88005f90 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:48.626 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.626+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde90016030 con 0x7fde98106650 2024-03-21T13:29:48.803 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:48.802+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"} v 0) v1 -- 0x7fde9806a3b0 con 0x7fde98106650 2024-03-21T13:29:48.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:29:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:29:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:48 smithi152 ceph-mon[34051]: pgmap v1241: 176 pgs: 176 active+clean; 732 KiB data, 874 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-21T13:29:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:29:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:48 smithi152 ceph-mon[34051]: osdmap e775: 8 total, 8 up, 8 in 2024-03-21T13:29:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:48 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2747277724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:48.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:48 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:48.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[39209]: pgmap v1241: 176 pgs: 176 active+clean; 732 KiB data, 874 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-21T13:29:48.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[39209]: osdmap e775: 8 total, 8 up, 8 in 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2747277724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[30095]: pgmap v1241: 176 pgs: 176 active+clean; 732 KiB data, 874 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[30095]: osdmap e775: 8 total, 8 up, 8 in 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2747277724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:48.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:48 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:49.411 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:49.411+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]=0 enabled application 'rados' on pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' v777) v1 ==== 213+0+0 (secure 0 0 0) 0x7fde9005f700 con 0x7fde98106650 2024-03-21T13:29:49.529 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:49.528+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"} v 0) v1 -- 0x7fde981089a0 con 0x7fde98106650 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]': finished 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: osdmap e776: 8 total, 8 up, 8 in 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2747277724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]: dispatch 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]': finished 2024-03-21T13:29:49.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:49 smithi152 ceph-mon[34051]: osdmap e777: 8 total, 8 up, 8 in 2024-03-21T13:29:49.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]': finished 2024-03-21T13:29:49.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: osdmap e776: 8 total, 8 up, 8 in 2024-03-21T13:29:49.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2747277724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:49.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]: dispatch 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]': finished 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[30095]: osdmap e777: 8 total, 8 up, 8 in 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]': finished 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: osdmap e776: 8 total, 8 up, 8 in 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2747277724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pg_num": 12}]: dispatch 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]: dispatch 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]': finished 2024-03-21T13:29:49.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:49 smithi089 ceph-mon[39209]: osdmap e777: 8 total, 8 up, 8 in 2024-03-21T13:29:50.426 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.426+0000 7fde877fe640 1 -- 172.21.15.89:0/4107355026 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]=0 enabled application 'rados' on pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' v778) v1 ==== 213+0+0 (secure 0 0 0) 0x7fde90063f30 con 0x7fde98106650 2024-03-21T13:29:50.427 INFO:tasks.workunit.client.0.smithi089.stderr:enabled application 'rados' on pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.430+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fde8006e2b0 msgr2=0x7fde80070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.430+0000 7fde9ce15640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fde8006e2b0 0x7fde80070770 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fde88009bd0 tx=0x7fde88005f90 comp rx=0 tx=0).stop 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.430+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98106650 msgr2=0x7fde98196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.430+0000 7fde9ce15640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98106650 0x7fde98196b10 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fde9000eaa0 tx=0x7fde9000ef60 comp rx=0 tx=0).stop 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.431+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 shutdown_connections 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.431+0000 7fde9ce15640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7fde8006e2b0 0x7fde80070770 unknown :-1 s=CLOSED pgs=3471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.431+0000 7fde9ce15640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde98106650 0x7fde98196b10 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.431+0000 7fde9ce15640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde98100f50 0x7fde981965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.432 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.431+0000 7fde9ce15640 1 --2- 172.21.15.89:0/4107355026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde98100590 0x7fde98196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.433 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.431+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 >> 172.21.15.89:0/4107355026 conn(0x7fde98077720 msgr2=0x7fde980ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:50.433 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.432+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 shutdown_connections 2024-03-21T13:29:50.433 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.432+0000 7fde9ce15640 1 -- 172.21.15.89:0/4107355026 wait complete. 2024-03-21T13:29:50.444 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota cf6754ed-1d53-435e-bb0e-835c0a37d941 max_objects 10 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 -- 172.21.15.89:0/3767806694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0101670 msgr2=0x7ff9a0101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/3767806694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0101670 0x7ff9a0101af0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7ff994009890 tx=0x7ff99402bb70 comp rx=0 tx=0).stop 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 -- 172.21.15.89:0/3767806694 shutdown_connections 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/3767806694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9a0102030 0x7ff9a010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/3767806694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0101670 0x7ff9a0101af0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/3767806694 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 0x7ff9a0107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 -- 172.21.15.89:0/3767806694 >> 172.21.15.89:0/3767806694 conn(0x7ff9a00fd160 msgr2=0x7ff9a00ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 -- 172.21.15.89:0/3767806694 shutdown_connections 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 -- 172.21.15.89:0/3767806694 wait complete. 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.531+0000 7ff9a55fe640 1 Processor -- start 2024-03-21T13:29:50.532 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff9a55fe640 1 -- start start 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff9a55fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9a0101670 0x7ff9a0197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff9a55fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0102030 0x7ff9a0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff9a55fe640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 0x7ff9a019bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff9a55fe640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a0110f60 con 0x7ff9a0102030 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff9a55fe640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a0110de0 con 0x7ff9a0101670 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff9a55fe640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a01110e0 con 0x7ff9a0107680 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 0x7ff9a019bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:50.535 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0102030 0x7ff9a0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 0x7ff9a019bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:49100/0 (socket says 172.21.15.89:49100) 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 -- 172.21.15.89:0/4077615779 learned_addr learned my addr 172.21.15.89:0/4077615779 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0102030 0x7ff9a0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56672/0 (socket says 172.21.15.89:56672) 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99effd640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9a0101670 0x7ff9a0197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 -- 172.21.15.89:0/4077615779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9a0101670 msgr2=0x7ff9a0197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9a0101670 0x7ff9a0197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 -- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0102030 msgr2=0x7ff9a0197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0102030 0x7ff9a0197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 -- 172.21.15.89:0/4077615779 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a019c510 con 0x7ff9a0107680 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99e7fc640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0102030 0x7ff9a0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99effd640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9a0101670 0x7ff9a0197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.532+0000 7ff99f7fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 0x7ff9a019bde0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7ff99800eaa0 tx=0x7ff99800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.533+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99800cdb0 con 0x7ff9a0107680 2024-03-21T13:29:50.536 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.533+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff998005020 con 0x7ff9a0107680 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.533+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a019c800 con 0x7ff9a0107680 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.533+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980116d0 con 0x7ff9a0107680 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.533+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a01a4090 con 0x7ff9a0107680 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.534+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a00689d0 con 0x7ff9a0107680 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.534+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7ff998004a40 con 0x7ff9a0107680 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.534+0000 7ff987fff640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff98806e2b0 0x7ff988070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.535+0000 7ff99effd640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff98806e2b0 0x7ff988070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.535+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(778..778 src has 1..778) v4 ==== 9551+0+0 (secure 0 0 0) 0x7ff99808f280 con 0x7ff9a0107680 2024-03-21T13:29:50.537 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.535+0000 7ff99effd640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff98806e2b0 0x7ff988070770 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7ff990005fd0 tx=0x7ff990009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:29:50.540 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.539+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff998016030 con 0x7ff9a0107680 2024-03-21T13:29:50.719 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:50.719+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7ff9a0109be0 con 0x7ff9a0107680 2024-03-21T13:29:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:50 smithi152 ceph-mon[34051]: pgmap v1244: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 874 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:50 smithi152 ceph-mon[34051]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:29:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]: dispatch 2024-03-21T13:29:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:50 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]': finished 2024-03-21T13:29:50.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:50 smithi152 ceph-mon[34051]: osdmap e778: 8 total, 8 up, 8 in 2024-03-21T13:29:50.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[30095]: pgmap v1244: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 874 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:50.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[30095]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:29:50.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]: dispatch 2024-03-21T13:29:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]': finished 2024-03-21T13:29:50.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[30095]: osdmap e778: 8 total, 8 up, 8 in 2024-03-21T13:29:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[39209]: pgmap v1244: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 874 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[39209]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:29:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]: dispatch 2024-03-21T13:29:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4107355026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "app": "rados"}]': finished 2024-03-21T13:29:50.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:50 smithi089 ceph-mon[39209]: osdmap e778: 8 total, 8 up, 8 in 2024-03-21T13:29:51.443 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:51.443+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool cf6754ed-1d53-435e-bb0e-835c0a37d941 v779) v1 ==== 223+0+0 (secure 0 0 0) 0x7ff99805f700 con 0x7ff9a0107680 2024-03-21T13:29:51.552 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:51.552+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7ff9a0109650 con 0x7ff9a0107680 2024-03-21T13:29:51.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:51 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4077615779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:51.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:51 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:51.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:51 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4077615779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:51.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:51 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:51.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:51 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4077615779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:51.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:51 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:52.450 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.449+0000 7ff987fff640 1 -- 172.21.15.89:0/4077615779 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool cf6754ed-1d53-435e-bb0e-835c0a37d941 v780) v1 ==== 223+0+0 (secure 0 0 0) 0x7ff998063f30 con 0x7ff9a0107680 2024-03-21T13:29:52.450 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_objects = 10 for pool cf6754ed-1d53-435e-bb0e-835c0a37d941 2024-03-21T13:29:52.453 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff98806e2b0 msgr2=0x7ff988070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:52.453 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff98806e2b0 0x7ff988070770 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7ff990005fd0 tx=0x7ff990009260 comp rx=0 tx=0).stop 2024-03-21T13:29:52.453 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 msgr2=0x7ff9a019bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:29:52.453 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 0x7ff9a019bde0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7ff99800eaa0 tx=0x7ff99800ef60 comp rx=0 tx=0).stop 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 shutdown_connections 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff98806e2b0 0x7ff988070770 unknown :-1 s=CLOSED pgs=3472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9a0107680 0x7ff9a019bde0 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9a0102030 0x7ff9a0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 --2- 172.21.15.89:0/4077615779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9a0101670 0x7ff9a0197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 >> 172.21.15.89:0/4077615779 conn(0x7ff9a00fd160 msgr2=0x7ff9a00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 shutdown_connections 2024-03-21T13:29:52.454 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:29:52.453+0000 7ff9a55fe640 1 -- 172.21.15.89:0/4077615779 wait complete. 2024-03-21T13:29:52.464 INFO:tasks.workunit.client.0.smithi089.stderr:+ sleep 30 2024-03-21T13:29:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:52 smithi152 ceph-mon[34051]: pgmap v1247: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:29:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:52 smithi152 ceph-mon[34051]: osdmap e779: 8 total, 8 up, 8 in 2024-03-21T13:29:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:52 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4077615779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:52.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:52 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:52.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[30095]: pgmap v1247: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:52.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[30095]: osdmap e779: 8 total, 8 up, 8 in 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4077615779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[39209]: pgmap v1247: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[39209]: osdmap e779: 8 total, 8 up, 8 in 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4077615779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:52.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:52 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]: dispatch 2024-03-21T13:29:53.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:29:53.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:53 smithi152 ceph-mon[34051]: osdmap e780: 8 total, 8 up, 8 in 2024-03-21T13:29:53.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:29:53.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:53 smithi089 ceph-mon[30095]: osdmap e780: 8 total, 8 up, 8 in 2024-03-21T13:29:53.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "field": "max_objects", "val": "10"}]': finished 2024-03-21T13:29:53.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:53 smithi089 ceph-mon[39209]: osdmap e780: 8 total, 8 up, 8 in 2024-03-21T13:29:54.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:54 smithi152 ceph-mon[34051]: pgmap v1250: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:54.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:54 smithi089 ceph-mon[30095]: pgmap v1250: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:54.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:54 smithi089 ceph-mon[39209]: pgmap v1250: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:29:55.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:55 smithi152 ceph-mon[34051]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:29:55.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:55 smithi089 ceph-mon[30095]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:29:55.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:55 smithi089 ceph-mon[39209]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T13:29:56.722 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:56 smithi152 ceph-mon[34051]: pgmap v1251: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-21T13:29:56.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:56 smithi089 ceph-mon[30095]: pgmap v1251: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-21T13:29:56.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:56 smithi089 ceph-mon[39209]: pgmap v1251: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-21T13:29:57.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:29:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:29:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:29:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:58 smithi152 ceph-mon[34051]: pgmap v1252: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-21T13:29:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:58 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:58.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:58.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:29:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:29:58.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:58 smithi089 ceph-mon[30095]: pgmap v1252: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-21T13:29:58.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:58 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:58.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:58 smithi089 ceph-mon[39209]: pgmap v1252: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-21T13:29:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:58 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:29:58.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:29:59.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:29:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:59.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:29:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:29:59.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:29:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:00 smithi152 ceph-mon[34051]: pgmap v1253: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:30:00.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:00 smithi152 ceph-mon[34051]: overall HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-21T13:30:00.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:00 smithi089 ceph-mon[39209]: pgmap v1253: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:30:00.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:00 smithi089 ceph-mon[39209]: overall HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-21T13:30:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:00 smithi089 ceph-mon[30095]: pgmap v1253: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:30:00.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:00 smithi089 ceph-mon[30095]: overall HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-21T13:30:02.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:02 smithi152 ceph-mon[34051]: pgmap v1254: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:30:02.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:02 smithi089 ceph-mon[30095]: pgmap v1254: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:30:02.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:02 smithi089 ceph-mon[39209]: pgmap v1254: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:30:04.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:04 smithi152 ceph-mon[34051]: pgmap v1255: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-21T13:30:04.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:04 smithi089 ceph-mon[30095]: pgmap v1255: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-21T13:30:04.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:04 smithi089 ceph-mon[39209]: pgmap v1255: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-21T13:30:06.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:06 smithi152 ceph-mon[34051]: pgmap v1256: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:06.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:06 smithi089 ceph-mon[30095]: pgmap v1256: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:06.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:06 smithi089 ceph-mon[39209]: pgmap v1256: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:07.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:30:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:30:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:30:08.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:08 smithi152 ceph-mon[34051]: pgmap v1257: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:08.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:30:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:30:08.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:08 smithi089 ceph-mon[30095]: pgmap v1257: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:08.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:08 smithi089 ceph-mon[39209]: pgmap v1257: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:09.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:09.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:09.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:10.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:10 smithi152 ceph-mon[34051]: pgmap v1258: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:10.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:10 smithi089 ceph-mon[39209]: pgmap v1258: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:10.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:10 smithi089 ceph-mon[30095]: pgmap v1258: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:12.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:12 smithi152 ceph-mon[34051]: pgmap v1259: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:12.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:12 smithi089 ceph-mon[39209]: pgmap v1259: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:12.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:12 smithi089 ceph-mon[30095]: pgmap v1259: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:13.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:13.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:13.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:14.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:14 smithi152 ceph-mon[34051]: pgmap v1260: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:14.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:14 smithi089 ceph-mon[30095]: pgmap v1260: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:14.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:14 smithi089 ceph-mon[39209]: pgmap v1260: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:16.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:16 smithi152 ceph-mon[34051]: pgmap v1261: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:16.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:16 smithi089 ceph-mon[30095]: pgmap v1261: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:16.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:16 smithi089 ceph-mon[39209]: pgmap v1261: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:30:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:30:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:30:18.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:18 smithi152 ceph-mon[34051]: pgmap v1262: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:18.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:30:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:30:18.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:18 smithi089 ceph-mon[30095]: pgmap v1262: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:18.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:18 smithi089 ceph-mon[39209]: pgmap v1262: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:19.820 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:19.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:19.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:20.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:20 smithi152 ceph-mon[34051]: pgmap v1263: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:20.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:20 smithi089 ceph-mon[30095]: pgmap v1263: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:20.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:20 smithi089 ceph-mon[39209]: pgmap v1263: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:22.466 INFO:tasks.workunit.client.0.smithi089.stderr:++ seq 1 10 2024-03-21T13:30:22.467 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.467 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj1 /etc/passwd 2024-03-21T13:30:22.507 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.507 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj2 /etc/passwd 2024-03-21T13:30:22.545 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.545 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj3 /etc/passwd 2024-03-21T13:30:22.582 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.582 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj4 /etc/passwd 2024-03-21T13:30:22.617 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.617 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj5 /etc/passwd 2024-03-21T13:30:22.652 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.652 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj6 /etc/passwd 2024-03-21T13:30:22.687 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.687 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj7 /etc/passwd 2024-03-21T13:30:22.721 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.721 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj8 /etc/passwd 2024-03-21T13:30:22.755 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.755 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj9 /etc/passwd 2024-03-21T13:30:22.790 INFO:tasks.workunit.client.0.smithi089.stderr:+ for f in `seq 1 10` 2024-03-21T13:30:22.790 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p cf6754ed-1d53-435e-bb0e-835c0a37d941 put obj10 /etc/passwd 2024-03-21T13:30:22.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:22 smithi152 ceph-mon[34051]: pgmap v1264: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:22.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:22 smithi089 ceph-mon[30095]: pgmap v1264: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:22.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:22 smithi089 ceph-mon[39209]: pgmap v1264: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:22.825 INFO:tasks.workunit.client.0.smithi089.stderr:+ sleep 30 2024-03-21T13:30:24.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:24 smithi152 ceph-mon[34051]: pgmap v1265: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:24.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:24 smithi089 ceph-mon[30095]: pgmap v1265: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:24.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:24 smithi089 ceph-mon[39209]: pgmap v1265: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:26.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:26 smithi152 ceph-mon[34051]: pgmap v1266: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:26.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:26 smithi089 ceph-mon[39209]: pgmap v1266: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:26.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:26 smithi089 ceph-mon[30095]: pgmap v1266: 188 pgs: 188 active+clean; 744 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:27.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:30:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:30:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:30:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:28 smithi152 ceph-mon[34051]: pgmap v1267: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-21T13:30:28.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:28.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:30:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:30:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:28 smithi089 ceph-mon[30095]: pgmap v1267: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-21T13:30:28.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:28 smithi089 ceph-mon[39209]: pgmap v1267: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-21T13:30:28.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:29.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:29.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:29.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:30 smithi152 ceph-mon[34051]: pgmap v1268: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-21T13:30:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:30 smithi152 ceph-mon[34051]: pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' is full (reached quota's max_objects: 10) 2024-03-21T13:30:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:30 smithi152 ceph-mon[34051]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:30:30.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:30 smithi152 ceph-mon[34051]: osdmap e781: 8 total, 8 up, 8 in 2024-03-21T13:30:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[30095]: pgmap v1268: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-21T13:30:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[30095]: pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' is full (reached quota's max_objects: 10) 2024-03-21T13:30:30.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[30095]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:30:30.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[30095]: osdmap e781: 8 total, 8 up, 8 in 2024-03-21T13:30:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[39209]: pgmap v1268: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-21T13:30:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[39209]: pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' is full (reached quota's max_objects: 10) 2024-03-21T13:30:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[39209]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-21T13:30:30.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:30 smithi089 ceph-mon[39209]: osdmap e781: 8 total, 8 up, 8 in 2024-03-21T13:30:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:32 smithi152 ceph-mon[34051]: pgmap v1270: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:32.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:32 smithi089 ceph-mon[39209]: pgmap v1270: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:32.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:32 smithi089 ceph-mon[30095]: pgmap v1270: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:34.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:34 smithi152 ceph-mon[34051]: pgmap v1271: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:34.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:34 smithi089 ceph-mon[39209]: pgmap v1271: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:34.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:34 smithi089 ceph-mon[30095]: pgmap v1271: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:36.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:36 smithi152 ceph-mon[34051]: pgmap v1272: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:36.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:36 smithi089 ceph-mon[30095]: pgmap v1272: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:36.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:36 smithi089 ceph-mon[39209]: pgmap v1272: 188 pgs: 188 active+clean; 859 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-21T13:30:37.825 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:30:37 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:30:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:30:38.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:38 smithi152 ceph-mon[34051]: pgmap v1273: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:30:38.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:30:38 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:30:38.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:38 smithi089 ceph-mon[30095]: pgmap v1273: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:30:38.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:38 smithi089 ceph-mon[39209]: pgmap v1273: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:30:39.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:39 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:39.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:39 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:39.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:39 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:40.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:40 smithi152 ceph-mon[34051]: pgmap v1274: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:30:40.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:40 smithi089 ceph-mon[39209]: pgmap v1274: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:30:40.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:40 smithi089 ceph-mon[30095]: pgmap v1274: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:30:42.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:42 smithi152 ceph-mon[34051]: pgmap v1275: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:30:42.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:42 smithi089 ceph-mon[30095]: pgmap v1275: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:30:42.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:42 smithi089 ceph-mon[39209]: pgmap v1275: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:30:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:30:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:43 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:30:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:30:43.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:43 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:30:43.822 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:43 smithi152 ceph-mon[34051]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:30:43.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[30095]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:30:43.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:30:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:30:43.826 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:43 smithi089 ceph-mon[39209]: from='mgr.24458 ' entity='mgr.y' 2024-03-21T13:30:44.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:44 smithi152 ceph-mon[34051]: pgmap v1276: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:44.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:44 smithi089 ceph-mon[30095]: pgmap v1276: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:44.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:44 smithi089 ceph-mon[39209]: pgmap v1276: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:46.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:46 smithi152 ceph-mon[34051]: pgmap v1277: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:46.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:46 smithi089 ceph-mon[39209]: pgmap v1277: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:46.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:46 smithi089 ceph-mon[30095]: pgmap v1277: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:47.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:30:47 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:30:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:30:49.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:48 smithi152 ceph-mon[34051]: pgmap v1278: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:49.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:30:48 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:30:49.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:48 smithi089 ceph-mon[30095]: pgmap v1278: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:49.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:48 smithi089 ceph-mon[39209]: pgmap v1278: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:50.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:49 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:50.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:49 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:50.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:49 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:30:51.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:50 smithi152 ceph-mon[34051]: pgmap v1279: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:51.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:50 smithi089 ceph-mon[30095]: pgmap v1279: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:51.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:50 smithi089 ceph-mon[39209]: pgmap v1279: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:52.826 INFO:tasks.workunit.client.0.smithi089.stderr:+ rados -p f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a put threemore /etc/passwd 2024-03-21T13:30:52.868 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a max_bytes 0 2024-03-21T13:30:52.958 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 -- 172.21.15.89:0/1858480477 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff80410bc50 msgr2=0x7ff80410e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:52.958 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 --2- 172.21.15.89:0/1858480477 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff80410bc50 0x7ff80410e0f0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009be0 tx=0x7ff7fc02c010 comp rx=0 tx=0).stop 2024-03-21T13:30:52.958 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 -- 172.21.15.89:0/1858480477 shutdown_connections 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 --2- 172.21.15.89:0/1858480477 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff80410bc50 0x7ff80410e0f0 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 --2- 172.21.15.89:0/1858480477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff804103230 0x7ff80410b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 --2- 172.21.15.89:0/1858480477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff804102910 0x7ff804102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 -- 172.21.15.89:0/1858480477 >> 172.21.15.89:0/1858480477 conn(0x7ff8040776c0 msgr2=0x7ff804077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 -- 172.21.15.89:0/1858480477 shutdown_connections 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.955+0000 7ff80882d640 1 -- 172.21.15.89:0/1858480477 wait complete. 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 Processor -- start 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 -- start start 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff804102910 0x7ff804197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff804103230 0x7ff804197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:52.959 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff80410bc50 0x7ff80419bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804110f20 con 0x7ff80410bc50 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804110da0 con 0x7ff804103230 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80882d640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8041110a0 con 0x7ff804102910 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80259c640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff804102910 0x7ff804197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80259c640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff804102910 0x7ff804197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:48292/0 (socket says 172.21.15.89:48292) 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff80259c640 1 -- 172.21.15.89:0/2802692720 learned_addr learned my addr 172.21.15.89:0/2802692720 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.956+0000 7ff802d9d640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff80410bc50 0x7ff80419bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff804103230 0x7ff804197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80259c640 1 -- 172.21.15.89:0/2802692720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff804103230 msgr2=0x7ff804197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80259c640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff804103230 0x7ff804197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80259c640 1 -- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff80410bc50 msgr2=0x7ff80419bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80259c640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff80410bc50 0x7ff80419bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:52.960 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80259c640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80419c430 con 0x7ff804102910 2024-03-21T13:30:52.961 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80259c640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff804102910 0x7ff804197440 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400e9f0 tx=0x7ff7f400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:30:52.961 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f400cd80 con 0x7ff804102910 2024-03-21T13:30:52.961 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff7f40108b0 con 0x7ff804102910 2024-03-21T13:30:52.961 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80419c720 con 0x7ff804102910 2024-03-21T13:30:52.961 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4018830 con 0x7ff804102910 2024-03-21T13:30:52.961 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.957+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8041a3fb0 con 0x7ff804102910 2024-03-21T13:30:52.961 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.958+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff804068580 con 0x7ff804102910 2024-03-21T13:30:52.962 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.958+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7ff7f4010430 con 0x7ff804102910 2024-03-21T13:30:52.962 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.959+0000 7ff7fb7fe640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff7f006e2b0 0x7ff7f0070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:52.962 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.959+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(781..781 src has 1..781) v4 ==== 9551+0+0 (secure 0 0 0) 0x7ff7f408f470 con 0x7ff804102910 2024-03-21T13:30:52.962 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.959+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff7f006e2b0 0x7ff7f0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:52.962 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.959+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=782}) v3 -- 0x7ff7f007a740 con 0x7ff804102910 2024-03-21T13:30:52.962 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.959+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff7f006e2b0 0x7ff7f0070770 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0096c0 tx=0x7ff7ec009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:30:52.964 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:52.964+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff7f405f9b0 con 0x7ff804102910 2024-03-21T13:30:53.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:52 smithi152 ceph-mon[34051]: pgmap v1280: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:53.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:52 smithi089 ceph-mon[30095]: pgmap v1280: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:53.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:52 smithi089 ceph-mon[39209]: pgmap v1280: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:30:53.142 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:53.142+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7ff804106dd0 con 0x7ff804102910 2024-03-21T13:30:53.594 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:53.594+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 7 ==== osd_map(782..782 src has 1..782) v4 ==== 628+0+0 (secure 0 0 0) 0x7ff7f40599b0 con 0x7ff804102910 2024-03-21T13:30:53.595 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:53.594+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=783}) v3 -- 0x7ff7f007b2b0 con 0x7ff804102910 2024-03-21T13:30:53.597 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:53.597+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v782) v1 ==== 217+0+0 (secure 0 0 0) 0x7ff7f40641e0 con 0x7ff804102910 2024-03-21T13:30:53.707 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:53.707+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7ff8041069e0 con 0x7ff804102910 2024-03-21T13:30:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:53 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2802692720' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:54.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:53 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:54.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:53 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2802692720' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:54.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:53 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:53 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2802692720' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:54.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:53 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:54.601 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.601+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 9 ==== osd_map(783..783 src has 1..783) v4 ==== 628+0+0 (secure 0 0 0) 0x7ff7f408cb30 con 0x7ff804102910 2024-03-21T13:30:54.601 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.601+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=784}) v3 -- 0x7ff7f007b8f0 con 0x7ff804102910 2024-03-21T13:30:54.607 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.607+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/2802692720 <== mon.2 v2:172.21.15.89:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v783) v1 ==== 217+0+0 (secure 0 0 0) 0x7ff7f4010d20 con 0x7ff804102910 2024-03-21T13:30:54.608 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_bytes = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:30:54.611 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff7f006e2b0 msgr2=0x7ff7f0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:54.611 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff7f006e2b0 0x7ff7f0070770 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0096c0 tx=0x7ff7ec009260 comp rx=0 tx=0).stop 2024-03-21T13:30:54.611 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff804102910 msgr2=0x7ff804197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:54.611 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff804102910 0x7ff804197440 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400e9f0 tx=0x7ff7f400eeb0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.611 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 shutdown_connections 2024-03-21T13:30:54.612 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7ff7f006e2b0 0x7ff7f0070770 unknown :-1 s=CLOSED pgs=3484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.612 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff80410bc50 0x7ff80419bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.612 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff804103230 0x7ff804197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.612 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 --2- 172.21.15.89:0/2802692720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff804102910 0x7ff804197440 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.612 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 >> 172.21.15.89:0/2802692720 conn(0x7ff8040776c0 msgr2=0x7ff8040fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:30:54.612 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 shutdown_connections 2024-03-21T13:30:54.612 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.610+0000 7ff80882d640 1 -- 172.21.15.89:0/2802692720 wait complete. 2024-03-21T13:30:54.622 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool set-quota f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a max_objects 0 2024-03-21T13:30:54.711 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 -- 172.21.15.89:0/3335466464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 msgr2=0x7f5214100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:54.711 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 --2- 172.21.15.89:0/3335466464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f5214100a00 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5204009860 tx=0x7f520402bb70 comp rx=0 tx=0).stop 2024-03-21T13:30:54.711 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 -- 172.21.15.89:0/3335466464 shutdown_connections 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 --2- 172.21.15.89:0/3335466464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5214100f40 0x7f521410d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 --2- 172.21.15.89:0/3335466464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f5214100a00 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 --2- 172.21.15.89:0/3335466464 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5214106640 0x7f5214106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 -- 172.21.15.89:0/3335466464 >> 172.21.15.89:0/3335466464 conn(0x7f52140776c0 msgr2=0x7f5214077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 -- 172.21.15.89:0/3335466464 shutdown_connections 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 -- 172.21.15.89:0/3335466464 wait complete. 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 Processor -- start 2024-03-21T13:30:54.712 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.711+0000 7f521be9d640 1 -- start start 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f52141982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5214100f40 0x7f52141987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5214106640 0x7f52141a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521410fdb0 con 0x7f5214100580 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521410fc30 con 0x7f5214100f40 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521410ff30 con 0x7f5214106640 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f5219c39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f52141982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521a43a640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5214106640 0x7f52141a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f5219c39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f52141982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34622/0 (socket says 172.21.15.89:34622) 2024-03-21T13:30:54.715 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f5219c39640 1 -- 172.21.15.89:0/4106441729 learned_addr learned my addr 172.21.15.89:0/4106441729 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f5219438640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5214100f40 0x7f52141987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521a43a640 1 -- 172.21.15.89:0/4106441729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5214100f40 msgr2=0x7f52141987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521a43a640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5214100f40 0x7f52141987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521a43a640 1 -- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 msgr2=0x7f52141982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521a43a640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f52141982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521a43a640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52141a3860 con 0x7f5214106640 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f5219c39640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f52141982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521a43a640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5214106640 0x7f52141a3320 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f520c00ca30 tx=0x7f520c00cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c007d90 con 0x7f5214106640 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52141a3a30 con 0x7f5214106640 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f520c005020 con 0x7f5214106640 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c00f6b0 con 0x7f5214106640 2024-03-21T13:30:54.716 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.712+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52141a3f50 con 0x7f5214106640 2024-03-21T13:30:54.717 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.713+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214068580 con 0x7f5214106640 2024-03-21T13:30:54.717 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.714+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f520c004a40 con 0x7f5214106640 2024-03-21T13:30:54.717 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.714+0000 7f520affd640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f520006e2b0 0x7f5200070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:30:54.717 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.714+0000 7f5219c39640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f520006e2b0 0x7f5200070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:30:54.717 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.714+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(783..783 src has 1..783) v4 ==== 9551+0+0 (secure 0 0 0) 0x7f520c08f390 con 0x7f5214106640 2024-03-21T13:30:54.717 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.715+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=784}) v3 -- 0x7f520007a740 con 0x7f5214106640 2024-03-21T13:30:54.717 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.715+0000 7f5219c39640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f520006e2b0 0x7f5200070770 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7f52100096c0 tx=0x7f5210009260 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:30:54.719 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.719+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f520c014070 con 0x7f5214106640 2024-03-21T13:30:54.896 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:54.896+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f52141a41e0 con 0x7f5214106640 2024-03-21T13:30:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:54 smithi152 ceph-mon[34051]: pgmap v1281: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:30:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:54 smithi152 ceph-mon[34051]: osdmap e782: 8 total, 8 up, 8 in 2024-03-21T13:30:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:54 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/2802692720' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:55.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:54 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:55.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[30095]: pgmap v1281: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:55.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:30:55.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[30095]: osdmap e782: 8 total, 8 up, 8 in 2024-03-21T13:30:55.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/2802692720' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:55.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[39209]: pgmap v1281: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:30:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:30:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[39209]: osdmap e782: 8 total, 8 up, 8 in 2024-03-21T13:30:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/2802692720' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:55.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:54 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-21T13:30:55.605 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:55.605+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 7 ==== osd_map(784..784 src has 1..784) v4 ==== 628+0+0 (secure 0 0 0) 0x7f520c0c3740 con 0x7f5214106640 2024-03-21T13:30:55.605 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:55.605+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=785}) v3 -- 0x7f520007b2b0 con 0x7f5214106640 2024-03-21T13:30:55.610 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:55.610+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v784) v1 ==== 221+0+0 (secure 0 0 0) 0x7f520c05f810 con 0x7f5214106640 2024-03-21T13:30:55.718 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:55.718+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f5214061c80 con 0x7f5214106640 2024-03-21T13:30:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:30:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:55 smithi152 ceph-mon[34051]: osdmap e783: 8 total, 8 up, 8 in 2024-03-21T13:30:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:55 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4106441729' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:56.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:55 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:30:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[30095]: osdmap e783: 8 total, 8 up, 8 in 2024-03-21T13:30:56.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4106441729' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:56.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_bytes", "val": "0"}]': finished 2024-03-21T13:30:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[39209]: osdmap e783: 8 total, 8 up, 8 in 2024-03-21T13:30:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4106441729' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:56.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:55 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:56.613 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.613+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 9 ==== osd_map(785..785 src has 1..785) v4 ==== 628+0+0 (secure 0 0 0) 0x7f520c059810 con 0x7f5214106640 2024-03-21T13:30:56.614 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.613+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=786}) v3 -- 0x7f520007b720 con 0x7f5214106640 2024-03-21T13:30:56.618 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.617+0000 7f520affd640 1 -- 172.21.15.89:0/4106441729 <== mon.2 v2:172.21.15.89:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a v785) v1 ==== 221+0+0 (secure 0 0 0) 0x7f520c064040 con 0x7f5214106640 2024-03-21T13:30:56.618 INFO:tasks.workunit.client.0.smithi089.stderr:set-quota max_objects = 0 for pool f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f520006e2b0 msgr2=0x7f5200070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f520006e2b0 0x7f5200070770 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7f52100096c0 tx=0x7f5210009260 comp rx=0 tx=0).stop 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5214106640 msgr2=0x7f52141a3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5214106640 0x7f52141a3320 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f520c00ca30 tx=0x7f520c00cef0 comp rx=0 tx=0).stop 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 shutdown_connections 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f520006e2b0 0x7f5200070770 unknown :-1 s=CLOSED pgs=3485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5214106640 0x7f52141a3320 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5214100f40 0x7f52141987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:56.622 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 --2- 172.21.15.89:0/4106441729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214100580 0x7f52141982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:30:56.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 >> 172.21.15.89:0/4106441729 conn(0x7f52140776c0 msgr2=0x7f5214104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:30:56.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 shutdown_connections 2024-03-21T13:30:56.623 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:30:56.621+0000 7f521be9d640 1 -- 172.21.15.89:0/4106441729 wait complete. 2024-03-21T13:30:56.632 INFO:tasks.workunit.client.0.smithi089.stderr:+ sleep 30 2024-03-21T13:30:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:56 smithi152 ceph-mon[34051]: pgmap v1284: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:30:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:30:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:56 smithi152 ceph-mon[34051]: osdmap e784: 8 total, 8 up, 8 in 2024-03-21T13:30:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:56 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4106441729' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:56.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:56 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:57.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[30095]: pgmap v1284: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[30095]: osdmap e784: 8 total, 8 up, 8 in 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4106441729' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[39209]: pgmap v1284: 188 pgs: 188 active+clean; 859 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[39209]: osdmap e784: 8 total, 8 up, 8 in 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4106441729' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:57.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:56 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]: dispatch 2024-03-21T13:30:57.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:30:57 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:30:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:30:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:57 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:30:57.825 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:57 smithi089 ceph-mon[30095]: osdmap e785: 8 total, 8 up, 8 in 2024-03-21T13:30:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:57 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:30:57.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:57 smithi089 ceph-mon[39209]: osdmap e785: 8 total, 8 up, 8 in 2024-03-21T13:30:58.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:57 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "field": "max_objects", "val": "0"}]': finished 2024-03-21T13:30:58.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:57 smithi152 ceph-mon[34051]: osdmap e785: 8 total, 8 up, 8 in 2024-03-21T13:30:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:58 smithi152 ceph-mon[34051]: pgmap v1287: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:30:59.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:58 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:59.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:30:58 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:30:59.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:58 smithi089 ceph-mon[30095]: pgmap v1287: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:30:59.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:58 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:30:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:58 smithi089 ceph-mon[39209]: pgmap v1287: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-21T13:30:59.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:58 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:31:00.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:30:59 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:00.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:30:59 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:00.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:30:59 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:01.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:00 smithi152 ceph-mon[34051]: pgmap v1288: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 931 B/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-21T13:31:01.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:00 smithi089 ceph-mon[30095]: pgmap v1288: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 931 B/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-21T13:31:01.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:00 smithi089 ceph-mon[39209]: pgmap v1288: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 931 B/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-21T13:31:03.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:02 smithi152 ceph-mon[34051]: pgmap v1289: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-21T13:31:03.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:02 smithi089 ceph-mon[30095]: pgmap v1289: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-21T13:31:03.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:02 smithi089 ceph-mon[39209]: pgmap v1289: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-21T13:31:05.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:04 smithi152 ceph-mon[34051]: pgmap v1290: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:31:05.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:04 smithi089 ceph-mon[30095]: pgmap v1290: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:31:05.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:04 smithi089 ceph-mon[39209]: pgmap v1290: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-21T13:31:07.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:06 smithi152 ceph-mon[34051]: pgmap v1291: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-21T13:31:07.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:06 smithi089 ceph-mon[30095]: pgmap v1291: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-21T13:31:07.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:06 smithi089 ceph-mon[39209]: pgmap v1291: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-21T13:31:07.826 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:07 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:31:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:31:09.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:08 smithi152 ceph-mon[34051]: pgmap v1292: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-03-21T13:31:09.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:31:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:31:09.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:08 smithi089 ceph-mon[30095]: pgmap v1292: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-03-21T13:31:09.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:08 smithi089 ceph-mon[39209]: pgmap v1292: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-03-21T13:31:10.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:09 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:10.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:09 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:10.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:09 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:10.976 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:10 smithi089 ceph-mon[30095]: pgmap v1293: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:10.976 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:10 smithi089 ceph-mon[39209]: pgmap v1293: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:11.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:10 smithi152 ceph-mon[34051]: pgmap v1293: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:13.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:12 smithi152 ceph-mon[34051]: pgmap v1294: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:13.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:12 smithi089 ceph-mon[30095]: pgmap v1294: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:13.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:12 smithi089 ceph-mon[39209]: pgmap v1294: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:14.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:13 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:31:14.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:13 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:31:14.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:13 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:31:15.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:14 smithi152 ceph-mon[34051]: pgmap v1295: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:15.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:14 smithi089 ceph-mon[30095]: pgmap v1295: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:15.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:14 smithi089 ceph-mon[39209]: pgmap v1295: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:17.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:16 smithi152 ceph-mon[34051]: pgmap v1296: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:17.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:16 smithi089 ceph-mon[39209]: pgmap v1296: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:17.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:16 smithi089 ceph-mon[30095]: pgmap v1296: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:17.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:17 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:31:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:31:19.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:18 smithi152 ceph-mon[34051]: pgmap v1297: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:19.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:31:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:31:19.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:18 smithi089 ceph-mon[30095]: pgmap v1297: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:19.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:18 smithi089 ceph-mon[39209]: pgmap v1297: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:20.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:19 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:20.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:19 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:20.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:19 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:21.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:20 smithi152 ceph-mon[34051]: pgmap v1298: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:31:21.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:20 smithi089 ceph-mon[30095]: pgmap v1298: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:31:21.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:20 smithi089 ceph-mon[39209]: pgmap v1298: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:31:23.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:22 smithi152 ceph-mon[34051]: pgmap v1299: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:23.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:22 smithi089 ceph-mon[30095]: pgmap v1299: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:23.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:22 smithi089 ceph-mon[39209]: pgmap v1299: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:31:25.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:24 smithi152 ceph-mon[34051]: pgmap v1300: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:25.074 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:24 smithi089 ceph-mon[39209]: pgmap v1300: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:25.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:24 smithi089 ceph-mon[30095]: pgmap v1300: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:26.634 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool delete f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a --yes-i-really-really-mean-it 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.727+0000 7f722413a640 1 -- 172.21.15.89:0/3377177491 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 msgr2=0x7f721c107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.727+0000 7f722413a640 1 --2- 172.21.15.89:0/3377177491 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 0x7f721c107a50 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f7218009930 tx=0x7f721802bc30 comp rx=0 tx=0).stop 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.727+0000 7f722413a640 1 -- 172.21.15.89:0/3377177491 shutdown_connections 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.727+0000 7f722413a640 1 --2- 172.21.15.89:0/3377177491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f721c102020 0x7f721c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.727+0000 7f722413a640 1 --2- 172.21.15.89:0/3377177491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c101660 0x7f721c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.727+0000 7f722413a640 1 --2- 172.21.15.89:0/3377177491 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 0x7f721c107a50 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.727+0000 7f722413a640 1 -- 172.21.15.89:0/3377177491 >> 172.21.15.89:0/3377177491 conn(0x7f721c0fd130 msgr2=0x7f721c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:31:26.728 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 -- 172.21.15.89:0/3377177491 shutdown_connections 2024-03-21T13:31:26.729 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 -- 172.21.15.89:0/3377177491 wait complete. 2024-03-21T13:31:26.731 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 Processor -- start 2024-03-21T13:31:26.731 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 -- start start 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f721c101660 0x7f721c1974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c102020 0x7f721c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 0x7f721c19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c110f10 con 0x7f721c102020 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c110d90 con 0x7f721c101660 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f722413a640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c111090 con 0x7f721c107670 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f72216d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c102020 0x7f721c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.728+0000 7f72216d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c102020 0x7f721c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55854/0 (socket says 172.21.15.89:55854) 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72226d7640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 0x7f721c19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:26.732 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72216d5640 1 -- 172.21.15.89:0/4000309133 learned_addr learned my addr 172.21.15.89:0/4000309133 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f7221ed6640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f721c101660 0x7f721c1974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72216d5640 1 -- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 msgr2=0x7f721c19bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72216d5640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 0x7f721c19bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72216d5640 1 -- 172.21.15.89:0/4000309133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f721c101660 msgr2=0x7f721c1974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72216d5640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f721c101660 0x7f721c1974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72216d5640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c19c4e0 con 0x7f721c102020 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f72216d5640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c102020 0x7f721c197a30 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f720c00ea60 tx=0x7f720c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c00cde0 con 0x7f721c102020 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c19c7d0 con 0x7f721c102020 2024-03-21T13:31:26.733 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f720c0035f0 con 0x7f721c102020 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c0044a0 con 0x7f721c102020 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.729+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c1a4060 con 0x7f721c102020 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.730+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c068580 con 0x7f721c102020 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.731+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f720c004000 con 0x7f721c102020 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.731+0000 7f7212ffd640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f720806e2b0 0x7f7208070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.731+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(785..785 src has 1..785) v4 ==== 9551+0+0 (secure 0 0 0) 0x7f720c08f470 con 0x7f721c102020 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.731+0000 7f7221ed6640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f720806e2b0 0x7f7208070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:26.734 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.731+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=786}) v3 -- 0x7f720807a740 con 0x7f721c102020 2024-03-21T13:31:26.735 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.732+0000 7f7221ed6640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f720806e2b0 0x7f7208070770 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f7218009900 tx=0x7f7218005fd0 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:31:26.736 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.736+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f720c05bea0 con 0x7f721c102020 2024-03-21T13:31:26.913 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:26.913+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f721c109bd0 con 0x7f721c102020 2024-03-21T13:31:27.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:26 smithi152 ceph-mon[34051]: pgmap v1301: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:27.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:26 smithi089 ceph-mon[30095]: pgmap v1301: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:27.075 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:26 smithi089 ceph-mon[39209]: pgmap v1301: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:31:27.684 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.684+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]=0 pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' removed v786) v1 ==== 248+0+0 (secure 0 0 0) 0x7f720c05f8f0 con 0x7f721c102020 2024-03-21T13:31:27.686 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.686+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 8 ==== osd_map(786..786 src has 1..786) v4 ==== 296+0+0 (secure 0 0 0) 0x7f720c059840 con 0x7f721c102020 2024-03-21T13:31:27.686 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.686+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=787}) v3 -- 0x7f720807b400 con 0x7f721c102020 2024-03-21T13:31:27.800 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.800+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f721c109810 con 0x7f721c102020 2024-03-21T13:31:27.801 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.801+0000 7f7212ffd640 1 -- 172.21.15.89:0/4000309133 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]=0 pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' does not exist v786) v1 ==== 255+0+0 (secure 0 0 0) 0x7f720c064120 con 0x7f721c102020 2024-03-21T13:31:27.801 INFO:tasks.workunit.client.0.smithi089.stderr:pool 'f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a' does not exist 2024-03-21T13:31:27.803 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f720806e2b0 msgr2=0x7f7208070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:27.803 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f720806e2b0 0x7f7208070770 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f7218009900 tx=0x7f7218005fd0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.803 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c102020 msgr2=0x7f721c197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:27.803 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c102020 0x7f721c197a30 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f720c00ea60 tx=0x7f720c00ef20 comp rx=0 tx=0).stop 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 shutdown_connections 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f720806e2b0 0x7f7208070770 unknown :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f721c107670 0x7f721c19bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f721c102020 0x7f721c197a30 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 --2- 172.21.15.89:0/4000309133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f721c101660 0x7f721c1974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 >> 172.21.15.89:0/4000309133 conn(0x7f721c0fd130 msgr2=0x7f721c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 shutdown_connections 2024-03-21T13:31:27.804 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.803+0000 7f722413a640 1 -- 172.21.15.89:0/4000309133 wait complete. 2024-03-21T13:31:27.812 INFO:tasks.workunit.client.0.smithi089.stderr:+ ceph osd pool delete cf6754ed-1d53-435e-bb0e-835c0a37d941 cf6754ed-1d53-435e-bb0e-835c0a37d941 --yes-i-really-really-mean-it 2024-03-21T13:31:27.824 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:27 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:27.824 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:27 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: ::ffff:172.21.15.152 - - [21/Mar/2024:13:31:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-21T13:31:27.825 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:27 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:27.897 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.894+0000 7f0df7702640 1 -- 172.21.15.89:0/3128401412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0106640 msgr2=0x7f0df0106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.894+0000 7f0df7702640 1 --2- 172.21.15.89:0/3128401412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0106640 0x7f0df0106a20 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009900 tx=0x7f0dec02bc30 comp rx=0 tx=0).stop 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- 172.21.15.89:0/3128401412 shutdown_connections 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 --2- 172.21.15.89:0/3128401412 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0100f40 0x7f0df010d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 --2- 172.21.15.89:0/3128401412 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0df0100580 0x7f0df0100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 --2- 172.21.15.89:0/3128401412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0106640 0x7f0df0106a20 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- 172.21.15.89:0/3128401412 >> 172.21.15.89:0/3128401412 conn(0x7f0df00776c0 msgr2=0x7f0df0077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- 172.21.15.89:0/3128401412 shutdown_connections 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- 172.21.15.89:0/3128401412 wait complete. 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 Processor -- start 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- start start 2024-03-21T13:31:27.898 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0df0100580 0x7f0df01982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0100f40 0x7f0df0198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0106640 0x7f0df01a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df010fdf0 con 0x7f0df0100f40 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df010fc70 con 0x7f0df0100580 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df7702640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df010ff70 con 0x7f0df0106640 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df4c9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0100f40 0x7f0df0198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df5c9f640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0106640 0x7f0df01a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df5c9f640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0106640 0x7f0df01a3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:33336/0 (socket says 172.21.15.89:33336) 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df4c9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0100f40 0x7f0df0198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53024/0 (socket says 172.21.15.89:53024) 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.895+0000 7f0df5c9f640 1 -- 172.21.15.89:0/3297626616 learned_addr learned my addr 172.21.15.89:0/3297626616 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df549e640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0df0100580 0x7f0df01982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df5c9f640 1 -- 172.21.15.89:0/3297626616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0df0100580 msgr2=0x7f0df01982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df5c9f640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0df0100580 0x7f0df01982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df5c9f640 1 -- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0100f40 msgr2=0x7f0df0198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:27.899 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df5c9f640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0100f40 0x7f0df0198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df5c9f640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df01a38a0 con 0x7f0df0106640 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df4c9d640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0100f40 0x7f0df0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df5c9f640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0106640 0x7f0df01a3360 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f0de800efd0 tx=0x7f0de800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8003b70 con 0x7f0df0106640 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df01a3a70 con 0x7f0df0106640 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0de8003d10 con 0x7f0df0106640 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8010770 con 0x7f0df0106640 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.896+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df01a3f70 con 0x7f0df0106640 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.897+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df0068580 con 0x7f0df0106640 2024-03-21T13:31:27.900 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.898+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 22) v1 ==== 92187+0+0 (secure 0 0 0) 0x7f0de8004000 con 0x7f0df0106640 2024-03-21T13:31:27.901 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.898+0000 7f0de67fc640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0ddc06e2b0 0x7f0ddc070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T13:31:27.901 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.898+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(786..786 src has 1..786) v4 ==== 9163+0+0 (secure 0 0 0) 0x7f0de808f390 con 0x7f0df0106640 2024-03-21T13:31:27.901 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.898+0000 7f0df549e640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0ddc06e2b0 0x7f0ddc070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T13:31:27.901 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.898+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=787}) v3 -- 0x7f0ddc079f20 con 0x7f0df0106640 2024-03-21T13:31:27.901 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.898+0000 7f0df549e640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0ddc06e2b0 0x7f0ddc070770 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000c00 tx=0x7f0dec02c810 comp rx=0 tx=0).ready entity=mgr.24458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T13:31:27.903 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:27.903+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0de805bfe0 con 0x7f0df0106640 2024-03-21T13:31:28.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:27 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:28.081 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.081+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0df01a4260 con 0x7f0df0106640 2024-03-21T13:31:28.687 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.686+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 7 ==== osd_map(787..787 src has 1..787) v4 ==== 296+0+0 (secure 0 0 0) 0x7f0de808cca0 con 0x7f0df0106640 2024-03-21T13:31:28.687 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.686+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=788}) v3 -- 0x7f0ddc07a7b0 con 0x7f0df0106640 2024-03-21T13:31:28.691 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.690+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]=0 pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' removed v787) v1 ==== 248+0+0 (secure 0 0 0) 0x7f0de805fa30 con 0x7f0df0106640 2024-03-21T13:31:28.805 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.805+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0df01a44e0 con 0x7f0df0106640 2024-03-21T13:31:28.806 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.806+0000 7f0de67fc640 1 -- 172.21.15.89:0/3297626616 <== mon.2 v2:172.21.15.89:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]=0 pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' does not exist v787) v1 ==== 255+0+0 (secure 0 0 0) 0x7f0de8064260 con 0x7f0df0106640 2024-03-21T13:31:28.806 INFO:tasks.workunit.client.0.smithi089.stderr:pool 'cf6754ed-1d53-435e-bb0e-835c0a37d941' does not exist 2024-03-21T13:31:28.808 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0ddc06e2b0 msgr2=0x7f0ddc070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:28.808 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0ddc06e2b0 0x7f0ddc070770 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000c00 tx=0x7f0dec02c810 comp rx=0 tx=0).stop 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0106640 msgr2=0x7f0df01a3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0106640 0x7f0df01a3360 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f0de800efd0 tx=0x7f0de800c6a0 comp rx=0 tx=0).stop 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 shutdown_connections 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:6800/2642177467,v1:172.21.15.89:6801/2642177467] conn(0x7f0ddc06e2b0 0x7f0ddc070770 unknown :-1 s=CLOSED pgs=3487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0df0106640 0x7f0df01a3360 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0df0100f40 0x7f0df0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 --2- 172.21.15.89:0/3297626616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0df0100580 0x7f0df01982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 >> 172.21.15.89:0/3297626616 conn(0x7f0df00776c0 msgr2=0x7f0df0104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T13:31:28.809 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 shutdown_connections 2024-03-21T13:31:28.810 INFO:tasks.workunit.client.0.smithi089.stderr:2024-03-21T13:31:28.808+0000 7f0df7702640 1 -- 172.21.15.89:0/3297626616 wait complete. 2024-03-21T13:31:28.817 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2024-03-21T13:31:28.818 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2024-03-21T13:31:28.818 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-21T13:31:28.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-21T13:31:28.886 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-21T13:31:28.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-21T13:31:28.950 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[39209]: pgmap v1302: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-21T13:31:28.950 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:31:28.950 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[39209]: osdmap e786: 8 total, 8 up, 8 in 2024-03-21T13:31:28.950 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3297626616' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[39209]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[30095]: pgmap v1302: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[30095]: osdmap e786: 8 total, 8 up, 8 in 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3297626616' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:28.951 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:28 smithi089 ceph-mon[30095]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:31:29.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug there is no tcmu-runner data available 2024-03-21T13:31:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-mon[34051]: pgmap v1302: 188 pgs: 188 active+clean; 871 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-21T13:31:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:31:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-mon[34051]: osdmap e786: 8 total, 8 up, 8 in 2024-03-21T13:31:29.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/4000309133' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "pool2": "f755f5fe-9ba8-48a3-9eb1-e65cb7ca201a", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3297626616' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:29.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:28 smithi152 ceph-mon[34051]: from='mgr.24458 172.21.15.89:0/1566746215' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:31:29.194 DEBUG:teuthology.parallel:result is None 2024-03-21T13:31:29.195 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-21T13:31:29.262 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-21T13:31:29.262 DEBUG:teuthology.orchestra.run.smithi089:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-21T13:31:29.327 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-21T13:31:29.327 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-21T13:31:29.337 INFO:tasks.cephadm:Teardown begin 2024-03-21T13:31:29.337 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:31:29.394 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:31:29.429 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-21T13:31:29.429 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:31:29.449 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:31:29.486 INFO:tasks.cephadm:Stopping all daemons... 2024-03-21T13:31:29.486 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-21T13:31:29.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a 2024-03-21T13:31:29.805 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[39209]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:31:29.805 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:31:29.805 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[39209]: osdmap e787: 8 total, 8 up, 8 in 2024-03-21T13:31:29.805 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[39209]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[39209]: from='client.? 172.21.15.89:0/3297626616' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[39209]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 systemd[1]: Stopping Ceph mon.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[30095]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[30095]: osdmap e787: 8 total, 8 up, 8 in 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[30095]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:29.806 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[30095]: from='client.? 172.21.15.89:0/3297626616' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:29.807 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-mon[30095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:29 smithi152 ceph-mon[34051]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-21T13:31:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T13:31:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:29 smithi152 ceph-mon[34051]: osdmap e787: 8 total, 8 up, 8 in 2024-03-21T13:31:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:29 smithi152 ceph-mon[34051]: from='client.24419 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:31:30.071 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:29 smithi152 ceph-mon[34051]: from='client.? 172.21.15.89:0/3297626616' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:30.072 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:29 smithi152 ceph-mon[34051]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "pool2": "cf6754ed-1d53-435e-bb0e-835c0a37d941", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T13:31:30.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a[30072]: 2024-03-21T13:31:29.804+0000 7f3ee322d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:31:30.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a[30072]: 2024-03-21T13:31:29.804+0000 7f3ee322d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-21T13:31:30.074 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:29 smithi089 podman[151584]: 2024-03-21 13:31:29.876166146 +0000 UTC m=+0.202238915 container died 7c27706dcbfb3d96e18d3a1220b981f232d12ddc40388c94ba5f816d8cae4954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-21T13:31:30.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:30 smithi089 podman[151584]: 2024-03-21 13:31:30.285431013 +0000 UTC m=+0.611503781 container cleanup 7c27706dcbfb3d96e18d3a1220b981f232d12ddc40388c94ba5f816d8cae4954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T13:31:30.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Mar 21 13:31:30 smithi089 bash[151584]: ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-a 2024-03-21T13:31:30.608 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.a.service' 2024-03-21T13:31:30.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:30.648 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-21T13:31:30.648 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-21T13:31:30.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.c 2024-03-21T13:31:30.824 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:30 smithi089 systemd[1]: Stopping Ceph mon.c for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:30 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c[39186]: 2024-03-21T13:31:30.975+0000 7f27d4d60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:31:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:30 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c[39186]: 2024-03-21T13:31:30.975+0000 7f27d4d60700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-21T13:31:31.325 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:31 smithi089 podman[151719]: 2024-03-21 13:31:31.048045295 +0000 UTC m=+0.177309397 container died a1d69bee49df90eb3ed93ca944d37bfd2bb82acc021f4348493bd5b01b204904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0) 2024-03-21T13:31:31.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:30 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:31:30] ENGINE Bus STOPPING 2024-03-21T13:31:31.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:30 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:31:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:31:31.325 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:30 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:31:30] ENGINE Bus STOPPED 2024-03-21T13:31:31.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:30 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:31:30] ENGINE Bus STARTING 2024-03-21T13:31:31.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:31 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:31:31] ENGINE Serving on http://:::9283 2024-03-21T13:31:31.326 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:31 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: [21/Mar/2024:13:31:31] ENGINE Bus STARTED 2024-03-21T13:31:31.799 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:31 smithi089 podman[151719]: 2024-03-21 13:31:31.498805756 +0000 UTC m=+0.628069849 container cleanup a1d69bee49df90eb3ed93ca944d37bfd2bb82acc021f4348493bd5b01b204904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-21T13:31:31.800 INFO:journalctl@ceph.mon.c.smithi089.stdout:Mar 21 13:31:31 smithi089 bash[151719]: ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-c 2024-03-21T13:31:31.916 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.c.service' 2024-03-21T13:31:31.953 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:31.953 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-21T13:31:31.953 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-21T13:31:31.953 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.b 2024-03-21T13:31:32.321 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:32 smithi152 systemd[1]: Stopping Ceph mon.b for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:32 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b[34028]: 2024-03-21T13:31:32.350+0000 7f11a612c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:31:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:32 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b[34028]: 2024-03-21T13:31:32.350+0000 7f11a612c700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-21T13:31:32.821 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:32 smithi152 podman[73379]: 2024-03-21 13:31:32.423135435 +0000 UTC m=+0.274138485 container died b11d157d3e17d970cd9fd5fecbdd56fc20d939f109410ca081081eba9884cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T13:31:33.267 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:33 smithi152 podman[73379]: 2024-03-21 13:31:33.049237092 +0000 UTC m=+0.900240141 container cleanup b11d157d3e17d970cd9fd5fecbdd56fc20d939f109410ca081081eba9884cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T13:31:33.267 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:33 smithi152 bash[73379]: ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b 2024-03-21T13:31:33.518 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.b.service' 2024-03-21T13:31:33.541 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:33 smithi152 podman[73398]: 2024-03-21 13:31:33.267147865 +0000 UTC m=+0.844067111 container remove b11d157d3e17d970cd9fd5fecbdd56fc20d939f109410ca081081eba9884cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:31:33.542 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:33 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-21T13:31:33.542 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:33 smithi152 systemd[1]: Stopped Ceph mon.b for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:31:33.542 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 21 13:31:33 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@mon.b.service: Consumed 27.304s CPU time. 2024-03-21T13:31:33.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:33.574 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-21T13:31:33.574 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-21T13:31:33.574 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y 2024-03-21T13:31:33.946 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:33 smithi089 systemd[1]: Stopping Ceph mgr.y for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:34.228 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:33 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:31:33.946+0000 7f749b323700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:31:34.228 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:33 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y[30351]: 2024-03-21T13:31:33.946+0000 7f749b323700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-21T13:31:34.228 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:33 smithi089 podman[151861]: 2024-03-21 13:31:33.978873727 +0000 UTC m=+0.214828640 container died ca14242b4eecdc9c5c1c1294ce98baacc4ad9c13a3e7b2a41ccfea5a97322341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-21T13:31:34.529 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:34 smithi089 podman[151861]: 2024-03-21 13:31:34.302779554 +0000 UTC m=+0.538734468 container cleanup ca14242b4eecdc9c5c1c1294ce98baacc4ad9c13a3e7b2a41ccfea5a97322341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T13:31:34.529 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:34 smithi089 bash[151861]: ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y 2024-03-21T13:31:34.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-21T13:31:34.783 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:34 smithi089 podman[151880]: 2024-03-21 13:31:34.529690492 +0000 UTC m=+0.550098992 container remove ca14242b4eecdc9c5c1c1294ce98baacc4ad9c13a3e7b2a41ccfea5a97322341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T13:31:34.783 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:34 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y.service: Deactivated successfully. 2024-03-21T13:31:34.783 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:34 smithi089 systemd[1]: Stopped Ceph mgr.y for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:31:34.783 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 21 13:31:34 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.y.service: Consumed 1min 32.644s CPU time. 2024-03-21T13:31:34.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:34.795 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-21T13:31:34.796 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-21T13:31:34.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.x 2024-03-21T13:31:35.109 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:31:34 smithi152 systemd[1]: Stopping Ceph mgr.x for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:35.379 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:31:35 smithi152 podman[73513]: 2024-03-21 13:31:35.110376148 +0000 UTC m=+0.123176745 container died 6319992840fae99306b7d8429ec60d1038f592cdc6073596486019b2cb510db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T13:31:35.653 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:31:35 smithi152 podman[73513]: 2024-03-21 13:31:35.461192772 +0000 UTC m=+0.473993378 container cleanup 6319992840fae99306b7d8429ec60d1038f592cdc6073596486019b2cb510db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:31:35.653 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Mar 21 13:31:35 smithi152 bash[73513]: ceph-9c355834-e783-11ee-95cd-87774f69a715-mgr-x 2024-03-21T13:31:35.871 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-21T13:31:35.910 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:35.911 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-21T13:31:35.911 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-21T13:31:35.911 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.0 2024-03-21T13:31:36.226 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:35 smithi089 systemd[1]: Stopping Ceph osd.0 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:36.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:36 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0[43618]: 2024-03-21T13:31:36.226+0000 7fd8e7763700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:31:36.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:36 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0[43618]: 2024-03-21T13:31:36.226+0000 7fd8e7763700 -1 osd.0 787 *** Got signal Terminated *** 2024-03-21T13:31:36.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:36 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0[43618]: 2024-03-21T13:31:36.226+0000 7fd8e7763700 -1 osd.0 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:31:41.958 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:41 smithi089 podman[151992]: 2024-03-21 13:31:41.660084947 +0000 UTC m=+5.554155636 container died 23f8840be96c7ad453f8a2956535603724d544ce1f020d089b6147d03c965fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-21T13:31:42.275 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 podman[151992]: 2024-03-21 13:31:42.049094505 +0000 UTC m=+5.943165260 container cleanup 23f8840be96c7ad453f8a2956535603724d544ce1f020d089b6147d03c965fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T13:31:42.275 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 bash[151992]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0 2024-03-21T13:31:42.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 podman[152014]: 2024-03-21 13:31:42.275228098 +0000 UTC m=+0.609636642 container remove 23f8840be96c7ad453f8a2956535603724d544ce1f020d089b6147d03c965fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-21T13:31:42.894 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 podman[152110]: 2024-03-21 13:31:42.576024663 +0000 UTC m=+0.110054783 container create 92e9bfe3cd63791859abe085bef6ad58b56ec7866437575cc53b230bc1991ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:31:42.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 podman[152110]: 2024-03-21 13:31:42.483571861 +0000 UTC m=+0.017601982 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:31:42.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 podman[152110]: 2024-03-21 13:31:42.812339787 +0000 UTC m=+0.346369910 container init 92e9bfe3cd63791859abe085bef6ad58b56ec7866437575cc53b230bc1991ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-21T13:31:42.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 podman[152110]: 2024-03-21 13:31:42.818601839 +0000 UTC m=+0.352631952 container start 92e9bfe3cd63791859abe085bef6ad58b56ec7866437575cc53b230bc1991ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-21T13:31:42.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:42 smithi089 podman[152110]: 2024-03-21 13:31:42.895040326 +0000 UTC m=+0.429070467 container attach 92e9bfe3cd63791859abe085bef6ad58b56ec7866437575cc53b230bc1991ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-21T13:31:43.324 INFO:journalctl@ceph.osd.0.smithi089.stdout:Mar 21 13:31:43 smithi089 podman[152110]: 2024-03-21 13:31:43.113647817 +0000 UTC m=+0.647677920 container died 92e9bfe3cd63791859abe085bef6ad58b56ec7866437575cc53b230bc1991ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T13:31:43.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.0.service' 2024-03-21T13:31:43.773 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:43.773 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-21T13:31:43.773 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-21T13:31:43.773 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.1 2024-03-21T13:31:44.074 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:43 smithi089 systemd[1]: Stopping Ceph osd.1 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:44.363 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:44 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1[49745]: 2024-03-21T13:31:44.145+0000 7f7ab1a03700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:31:44.363 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:44 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1[49745]: 2024-03-21T13:31:44.145+0000 7f7ab1a03700 -1 osd.1 787 *** Got signal Terminated *** 2024-03-21T13:31:44.363 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:44 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1[49745]: 2024-03-21T13:31:44.145+0000 7f7ab1a03700 -1 osd.1 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:31:49.792 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:49 smithi089 podman[152250]: 2024-03-21 13:31:49.496519617 +0000 UTC m=+5.487731456 container died 877938a87a42adbb7ecb6f44abebbb9cc6a570bbf240b45a46a8b14e2bb70155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux ) 2024-03-21T13:31:50.074 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:49 smithi089 podman[152250]: 2024-03-21 13:31:49.792883631 +0000 UTC m=+5.784095469 container cleanup 877938a87a42adbb7ecb6f44abebbb9cc6a570bbf240b45a46a8b14e2bb70155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T13:31:50.075 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:49 smithi089 bash[152250]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1 2024-03-21T13:31:50.075 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:49 smithi089 podman[152276]: 2024-03-21 13:31:49.98566517 +0000 UTC m=+0.488503385 container remove 877938a87a42adbb7ecb6f44abebbb9cc6a570bbf240b45a46a8b14e2bb70155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:31:50.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:50 smithi089 podman[152374]: 2024-03-21 13:31:50.213890181 +0000 UTC m=+0.019112088 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:31:50.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:50 smithi089 podman[152374]: 2024-03-21 13:31:50.311427574 +0000 UTC m=+0.116649474 container create c369198e461e5a70b8801863570b59105396c91dfb040d5ae3b5caa29839cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-21T13:31:50.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:50 smithi089 podman[152374]: 2024-03-21 13:31:50.548135589 +0000 UTC m=+0.353357495 container init c369198e461e5a70b8801863570b59105396c91dfb040d5ae3b5caa29839cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:31:50.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:50 smithi089 podman[152374]: 2024-03-21 13:31:50.554200889 +0000 UTC m=+0.359422799 container start c369198e461e5a70b8801863570b59105396c91dfb040d5ae3b5caa29839cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T13:31:51.075 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:50 smithi089 podman[152374]: 2024-03-21 13:31:50.647553786 +0000 UTC m=+0.452775687 container attach c369198e461e5a70b8801863570b59105396c91dfb040d5ae3b5caa29839cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:31:51.075 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:50 smithi089 podman[152374]: 2024-03-21 13:31:50.844679587 +0000 UTC m=+0.649901488 container died c369198e461e5a70b8801863570b59105396c91dfb040d5ae3b5caa29839cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-21T13:31:51.400 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.1.service' 2024-03-21T13:31:51.427 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:51 smithi089 podman[152420]: 2024-03-21 13:31:51.376051159 +0000 UTC m=+0.522901750 container remove c369198e461e5a70b8801863570b59105396c91dfb040d5ae3b5caa29839cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T13:31:51.427 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:51 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-21T13:31:51.427 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:51 smithi089 systemd[1]: Stopped Ceph osd.1 for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:31:51.427 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 21 13:31:51 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.1.service: Consumed 43.284s CPU time. 2024-03-21T13:31:51.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:51.444 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-21T13:31:51.444 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-21T13:31:51.444 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.2 2024-03-21T13:31:51.823 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:51 smithi089 systemd[1]: Stopping Ceph osd.2 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:31:52.074 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2[55730]: 2024-03-21T13:31:51.822+0000 7fbb1feb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:31:52.074 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2[55730]: 2024-03-21T13:31:51.822+0000 7fbb1feb9700 -1 osd.2 787 *** Got signal Terminated *** 2024-03-21T13:31:52.075 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:51 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2[55730]: 2024-03-21T13:31:51.822+0000 7fbb1feb9700 -1 osd.2 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:31:57.554 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:57 smithi089 podman[152505]: 2024-03-21 13:31:57.245044269 +0000 UTC m=+5.583413657 container died f96ad318d2912edc7728a52275ed7a164c4e08dae004fcbb3298f310180eb6a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:31:57.820 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:57 smithi089 podman[152505]: 2024-03-21 13:31:57.644916532 +0000 UTC m=+5.983285920 container cleanup f96ad318d2912edc7728a52275ed7a164c4e08dae004fcbb3298f310180eb6a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:31:57.820 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:57 smithi089 bash[152505]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2 2024-03-21T13:31:58.074 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:57 smithi089 podman[152527]: 2024-03-21 13:31:57.821077071 +0000 UTC m=+0.570756396 container remove f96ad318d2912edc7728a52275ed7a164c4e08dae004fcbb3298f310180eb6a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-21T13:31:58.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:58 smithi089 podman[152624]: 2024-03-21 13:31:58.221955903 +0000 UTC m=+0.087224092 container create 01d663034611c8d4296d1696615d2da33ef9856dcf483ad27e8a28c03a764ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-21T13:31:58.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:58 smithi089 podman[152624]: 2024-03-21 13:31:58.153836494 +0000 UTC m=+0.019104617 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:31:58.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:58 smithi089 podman[152624]: 2024-03-21 13:31:58.424254827 +0000 UTC m=+0.289522958 container init 01d663034611c8d4296d1696615d2da33ef9856dcf483ad27e8a28c03a764ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:31:58.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:58 smithi089 podman[152624]: 2024-03-21 13:31:58.4296563 +0000 UTC m=+0.294924424 container start 01d663034611c8d4296d1696615d2da33ef9856dcf483ad27e8a28c03a764ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5) 2024-03-21T13:31:58.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:58 smithi089 podman[152624]: 2024-03-21 13:31:58.490585901 +0000 UTC m=+0.355854031 container attach 01d663034611c8d4296d1696615d2da33ef9856dcf483ad27e8a28c03a764ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:31:58.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Mar 21 13:31:58 smithi089 podman[152624]: 2024-03-21 13:31:58.721046532 +0000 UTC m=+0.586314660 container died 01d663034611c8d4296d1696615d2da33ef9856dcf483ad27e8a28c03a764ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:31:59.233 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.2.service' 2024-03-21T13:31:59.276 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:31:59.276 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-21T13:31:59.276 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-21T13:31:59.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.3 2024-03-21T13:31:59.577 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:31:59 smithi089 systemd[1]: Stopping Ceph osd.3 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:00.074 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:31:59 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3[62387]: 2024-03-21T13:31:59.607+0000 7fc59ad2a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:32:00.074 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:31:59 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3[62387]: 2024-03-21T13:31:59.607+0000 7fc59ad2a700 -1 osd.3 787 *** Got signal Terminated *** 2024-03-21T13:32:00.075 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:31:59 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3[62387]: 2024-03-21T13:31:59.607+0000 7fc59ad2a700 -1 osd.3 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:32:03.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:02 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:02.764+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:03.687 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:03 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:03.396+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:04.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:03 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:03.802+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:04.071 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:03 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:03.919+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:04.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:03 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:03.687+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:04.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:03 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:03.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:04.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:03 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:03.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused" 2024-03-21T13:32:04.682 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:04 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:04.350+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:05.071 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:04 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:04.906+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:05.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:04 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:04.803+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:05.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:04 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:04.683+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:05.297 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:05 smithi089 podman[152753]: 2024-03-21 13:32:05.032362019 +0000 UTC m=+5.535196878 container died 683093a9f3c2e481486311a348827bc13bf2ce86eb6dfc9fd694e2046f57f6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T13:32:05.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:05 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:05.313+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:05.574 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:05 smithi089 podman[152753]: 2024-03-21 13:32:05.404703693 +0000 UTC m=+5.907538552 container cleanup 683093a9f3c2e481486311a348827bc13bf2ce86eb6dfc9fd694e2046f57f6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-21T13:32:05.574 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:05 smithi089 bash[152753]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3 2024-03-21T13:32:06.014 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:05 smithi089 podman[152776]: 2024-03-21 13:32:05.664398467 +0000 UTC m=+0.628418814 container remove 683093a9f3c2e481486311a348827bc13bf2ce86eb6dfc9fd694e2046f57f6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:32:06.070 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:05 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:05.788+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:06.071 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:05 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:05.918+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:06.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:05 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:05.726+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:06.283 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:06 smithi089 podman[152870]: 2024-03-21 13:32:06.014968246 +0000 UTC m=+0.106799580 container create 000892c61e790793d500e660864d99e03dcc692bec7bd252796f3b2b230275a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS) 2024-03-21T13:32:06.283 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:06 smithi089 podman[152870]: 2024-03-21 13:32:05.926498238 +0000 UTC m=+0.018329586 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:32:06.283 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:06 smithi089 podman[152870]: 2024-03-21 13:32:06.192215175 +0000 UTC m=+0.284046515 container init 000892c61e790793d500e660864d99e03dcc692bec7bd252796f3b2b230275a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:32:06.283 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:06 smithi089 podman[152870]: 2024-03-21 13:32:06.197623812 +0000 UTC m=+0.289455143 container start 000892c61e790793d500e660864d99e03dcc692bec7bd252796f3b2b230275a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T13:32:06.283 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:06 smithi089 podman[152870]: 2024-03-21 13:32:06.283764664 +0000 UTC m=+0.375595997 container attach 000892c61e790793d500e660864d99e03dcc692bec7bd252796f3b2b230275a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-21T13:32:06.574 INFO:journalctl@ceph.osd.3.smithi089.stdout:Mar 21 13:32:06 smithi089 podman[152870]: 2024-03-21 13:32:06.496227927 +0000 UTC m=+0.588059253 container died 000892c61e790793d500e660864d99e03dcc692bec7bd252796f3b2b230275a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T13:32:06.724 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:06 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:06.346+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:07.071 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:06 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:06.937+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:07.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:06 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:06.723+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:07.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:06 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:06.836+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:07.078 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.3.service' 2024-03-21T13:32:07.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:32:07.118 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-21T13:32:07.118 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-21T13:32:07.118 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.4 2024-03-21T13:32:07.531 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:07 smithi152 systemd[1]: Stopping Ceph osd.4 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:07.532 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:07.375+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:07.821 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:07.712+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:07.821 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:07.531+0000 7f3e89aa7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:32:07.821 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:07.531+0000 7f3e89aa7700 -1 osd.4 787 *** Got signal Terminated *** 2024-03-21T13:32:07.821 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:07.531+0000 7f3e89aa7700 -1 osd.4 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:32:08.321 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:07.979+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:08.321 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:07 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:07.823+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:08.700 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:08.356+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:09.001 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:08.844+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:09.001 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:08 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:08.700+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:09.321 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:09 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:09.000+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:09.676 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:09 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:09.357+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:10.032 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:09 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:09.803+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:10.033 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:09 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:09.676+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:10.321 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:10 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:10.033+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:10.676 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:10 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:10.371+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:11.008 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:10 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:10.776+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:11.008 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:10 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:10.676+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:11.321 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:11.008+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:11.677 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:11.357+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:11.677 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:11.357+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:12.036 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:11.777+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:12.037 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:11.777+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:12.037 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:11.677+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:12.037 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:11 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:11.677+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:12.319 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:12.036+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:41.219626+0000 front 2024-03-21T13:31:41.219527+0000 (oldest deadline 2024-03-21T13:32:03.519424+0000) 2024-03-21T13:32:12.319 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4[39601]: 2024-03-21T13:32:12.036+0000 7f3e82093700 -1 osd.4 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.921027+0000 front 2024-03-21T13:31:46.921123+0000 (oldest deadline 2024-03-21T13:32:11.621042+0000) 2024-03-21T13:32:12.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:12.319+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:12.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:12.319+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:12.571 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:12.362Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.89:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.89:8765: connect: connection refused" 2024-03-21T13:32:12.572 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:12.365Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.89:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.89:8765: connect: connection refused" 2024-03-21T13:32:12.572 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:12.367Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.89:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.89:8765: connect: connection refused" 2024-03-21T13:32:12.572 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:12.367Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.89:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.89:8765: connect: connection refused" 2024-03-21T13:32:12.572 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:12.367Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.89:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.89:8765: connect: connection refused" 2024-03-21T13:32:12.944 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:12.768+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:12.944 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:12.768+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:12.944 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:12.683+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:12.944 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:12 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:12.683+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:13.196 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:12 smithi152 podman[73650]: 2024-03-21 13:32:12.944462627 +0000 UTC m=+5.628087624 container died 84670b57bb7b6af3d05404420a537e877d7790bd26ecda19783769621064447b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T13:32:13.571 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:13 smithi152 podman[73650]: 2024-03-21 13:32:13.287297286 +0000 UTC m=+5.970922276 container cleanup 84670b57bb7b6af3d05404420a537e877d7790bd26ecda19783769621064447b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T13:32:13.571 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:13 smithi152 bash[73650]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4 2024-03-21T13:32:13.572 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:13 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:13.318+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:13.572 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:13 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:13.318+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:13.903 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:13 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:13.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:13.903 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:13 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:13.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:13.903 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:13 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:13.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused" 2024-03-21T13:32:13.951 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:13 smithi152 podman[73672]: 2024-03-21 13:32:13.613272686 +0000 UTC m=+0.672493033 container remove 84670b57bb7b6af3d05404420a537e877d7790bd26ecda19783769621064447b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:32:13.951 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:13 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:13.816+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:13.951 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:13 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:13.816+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:13.951 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:13 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:13.639+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:13.951 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:13 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:13.639+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:14.274 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:13 smithi152 podman[73766]: 2024-03-21 13:32:13.854422213 +0000 UTC m=+0.018260520 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:32:14.274 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:13 smithi152 podman[73766]: 2024-03-21 13:32:13.955764294 +0000 UTC m=+0.119602643 container create d0a8d4409a357f7834912788f6cc4d4618dc4cc880b215de1c6386ddbf921a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-21T13:32:14.274 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:14 smithi152 podman[73766]: 2024-03-21 13:32:14.174636904 +0000 UTC m=+0.338475208 container init d0a8d4409a357f7834912788f6cc4d4618dc4cc880b215de1c6386ddbf921a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-21T13:32:14.274 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:14 smithi152 podman[73766]: 2024-03-21 13:32:14.179887468 +0000 UTC m=+0.343725776 container start d0a8d4409a357f7834912788f6cc4d4618dc4cc880b215de1c6386ddbf921a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T13:32:14.274 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:14 smithi152 podman[73766]: 2024-03-21 13:32:14.274539027 +0000 UTC m=+0.438377347 container attach d0a8d4409a357f7834912788f6cc4d4618dc4cc880b215de1c6386ddbf921a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-21T13:32:14.571 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 21 13:32:14 smithi152 podman[73766]: 2024-03-21 13:32:14.471244717 +0000 UTC m=+0.635083018 container died d0a8d4409a357f7834912788f6cc4d4618dc4cc880b215de1c6386ddbf921a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True) 2024-03-21T13:32:14.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:14 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:14.287+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:14.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:14 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:14.287+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:15.069 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:14 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:14.767+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:15.069 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:14 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:14.767+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:15.069 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:14 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:14.685+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:15.069 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:14 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:14.685+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:15.094 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.4.service' 2024-03-21T13:32:15.135 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:32:15.135 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-21T13:32:15.135 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-21T13:32:15.135 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.5 2024-03-21T13:32:15.321 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:15 smithi152 systemd[1]: Stopping Ceph osd.5 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:15.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:15.258+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:15.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:15.258+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:15.722 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:15.466+0000 7f9ddac85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:32:15.722 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:15.466+0000 7f9ddac85700 -1 osd.5 787 *** Got signal Terminated *** 2024-03-21T13:32:15.722 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:15.466+0000 7f9ddac85700 -1 osd.5 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:32:16.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:15.804+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:16.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:15.804+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:16.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:15.722+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:16.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:15 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:15.722+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:16.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:16 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:16.226+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:16.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:16 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:16.226+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:17.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:16 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:16.771+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:17.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:16 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:16.771+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:17.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:16 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:16.731+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:17.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:16 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:16.731+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:17.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:17.240+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:17.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:17.240+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:18.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:17.756+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:18.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:17.756+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:18.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:17.760+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:18.072 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:17 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:17.760+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:18.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:18.242+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:18.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:18.242+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:18.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:18.242+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:19.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:18.783+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:19.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:18.783+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:19.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:18.711+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:19.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:18 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:18.711+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:19.573 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:19 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:19.198+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:19.574 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:19 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:19.198+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:19.574 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:19 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:19.198+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:20.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:19 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:19.765+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:37.424769+0000 front 2024-03-21T13:31:37.424987+0000 (oldest deadline 2024-03-21T13:32:02.724965+0000) 2024-03-21T13:32:20.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:19 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5[45691]: 2024-03-21T13:32:19.765+0000 7f9dd3271700 -1 osd.5 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.226074+0000 front 2024-03-21T13:31:46.226165+0000 (oldest deadline 2024-03-21T13:32:11.526057+0000) 2024-03-21T13:32:20.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:19 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:19.669+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:20.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:19 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:19.669+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:20.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:20 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:20.236+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:20.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:20 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:20.236+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:20.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:20 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:20.236+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:21.071 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:20 smithi152 podman[73897]: 2024-03-21 13:32:20.869264513 +0000 UTC m=+5.510982880 container died cab6442d656df9e10d08b5b20baf2dd60d43bb071bef61ed69cc6eba69e7de52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-21T13:32:21.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:20 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:20.675+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:21.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:20 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:20.675+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:21.464 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:21 smithi152 podman[73897]: 2024-03-21 13:32:21.271972844 +0000 UTC m=+5.913691212 container cleanup cab6442d656df9e10d08b5b20baf2dd60d43bb071bef61ed69cc6eba69e7de52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-21T13:32:21.464 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:21 smithi152 bash[73897]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5 2024-03-21T13:32:21.464 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:21 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:21.275+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:21.464 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:21 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:21.275+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:21.465 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:21 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:21.275+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:21.821 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:21 smithi152 podman[73919]: 2024-03-21 13:32:21.464720259 +0000 UTC m=+0.593978871 container remove cab6442d656df9e10d08b5b20baf2dd60d43bb071bef61ed69cc6eba69e7de52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:32:21.822 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:21 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:21.646+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:21.822 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:21 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:21.646+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:22.151 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:21 smithi152 podman[74018]: 2024-03-21 13:32:21.791531485 +0000 UTC m=+0.019044806 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:32:22.151 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:21 smithi152 podman[74018]: 2024-03-21 13:32:21.907131333 +0000 UTC m=+0.134644636 container create e3ed12fc1e1e2257f58bc794a2c7fe30c1c068872a703744b58e9e39faf91f2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-21T13:32:22.151 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:22 smithi152 podman[74018]: 2024-03-21 13:32:22.151678747 +0000 UTC m=+0.379192064 container init e3ed12fc1e1e2257f58bc794a2c7fe30c1c068872a703744b58e9e39faf91f2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-21T13:32:22.448 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:22 smithi152 podman[74018]: 2024-03-21 13:32:22.157009312 +0000 UTC m=+0.384522617 container start e3ed12fc1e1e2257f58bc794a2c7fe30c1c068872a703744b58e9e39faf91f2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:32:22.448 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:22 smithi152 podman[74018]: 2024-03-21 13:32:22.234827408 +0000 UTC m=+0.462340724 container attach e3ed12fc1e1e2257f58bc794a2c7fe30c1c068872a703744b58e9e39faf91f2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T13:32:22.448 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:22 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:22.254+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:22.448 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:22 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:22.254+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:22.448 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:22 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:22.254+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:22.762 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 21 13:32:22 smithi152 podman[74018]: 2024-03-21 13:32:22.448829443 +0000 UTC m=+0.676342755 container died e3ed12fc1e1e2257f58bc794a2c7fe30c1c068872a703744b58e9e39faf91f2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True) 2024-03-21T13:32:22.763 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:22 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:22.691+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:22.763 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:22 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:22.691+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:22.763 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:22 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:22.691+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:23.056 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.5.service' 2024-03-21T13:32:23.097 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:32:23.098 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-21T13:32:23.098 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-21T13:32:23.098 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.6 2024-03-21T13:32:23.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:23 smithi152 systemd[1]: Stopping Ceph osd.6 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:23.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:23.239+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:23.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:23.239+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:23.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:23.239+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:23.728 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:23.417+0000 7f2b204e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:32:23.728 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:23.417+0000 7f2b204e2700 -1 osd.6 787 *** Got signal Terminated *** 2024-03-21T13:32:23.728 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:23.417+0000 7f2b204e2700 -1 osd.6 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:32:24.070 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:23.728+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:24.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:23.728+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:24.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:23 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:23.728+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:24.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:23 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:23.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused" 2024-03-21T13:32:24.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:23 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:23.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:24.075 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:23 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:23.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused" 2024-03-21T13:32:24.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:24 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:24.280+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:24.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:24 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:24.280+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:24.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:24 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:24.280+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:25.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:24 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:24.740+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:25.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:24 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:24.740+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:25.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:24 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:24.740+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:25.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:25 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:25.314+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:25.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:25 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:25.314+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:25.571 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:25 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:25.314+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:26.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:25 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:25.757+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:26.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:25 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:25.757+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:26.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:25 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:25.757+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:26.735 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:26 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:26.340+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:26.736 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:26 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:26.340+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:26.736 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:26 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:26.340+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:27.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:26 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:26.736+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:27.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:26 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:26.736+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:27.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:26 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:26.736+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:27.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:26 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:26.736+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:27.748 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:27.356+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:27.748 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:27.356+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:27.748 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:27.356+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:27.749 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:27.356+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:01.690489+0000 front 2024-03-21T13:32:01.690432+0000 (oldest deadline 2024-03-21T13:32:26.390355+0000) 2024-03-21T13:32:28.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:27.748+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:28.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:27.748+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:28.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:27.748+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:28.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:27.748+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:28.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:27 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:27.748+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:28.746 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:28.406+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:39.586907+0000 front 2024-03-21T13:31:39.587015+0000 (oldest deadline 2024-03-21T13:32:03.086949+0000) 2024-03-21T13:32:28.747 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:28.406+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:45.888634+0000 front 2024-03-21T13:31:45.888485+0000 (oldest deadline 2024-03-21T13:32:11.188505+0000) 2024-03-21T13:32:28.747 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:28.406+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:55.289500+0000 front 2024-03-21T13:31:55.289543+0000 (oldest deadline 2024-03-21T13:32:18.189390+0000) 2024-03-21T13:32:28.747 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6[51644]: 2024-03-21T13:32:28.406+0000 7f2b18ace700 -1 osd.6 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:01.690489+0000 front 2024-03-21T13:32:01.690432+0000 (oldest deadline 2024-03-21T13:32:26.390355+0000) 2024-03-21T13:32:29.024 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:28 smithi152 podman[74150]: 2024-03-21 13:32:28.749527729 +0000 UTC m=+5.434475193 container died 07df11cb75bbebaa42a19aaef6dd47cd94f7b9fb1e63d075c1b23ec6a749d9fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:32:29.024 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:28.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:29.024 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:28.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:29.024 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:28.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:29.024 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:28.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:29.024 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:28 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:28.746+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:29.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 podman[74150]: 2024-03-21 13:32:29.124078507 +0000 UTC m=+5.809025974 container cleanup 07df11cb75bbebaa42a19aaef6dd47cd94f7b9fb1e63d075c1b23ec6a749d9fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-21T13:32:29.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 bash[74150]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6 2024-03-21T13:32:29.626 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 podman[74171]: 2024-03-21 13:32:29.325096338 +0000 UTC m=+0.572486664 container remove 07df11cb75bbebaa42a19aaef6dd47cd94f7b9fb1e63d075c1b23ec6a749d9fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-21T13:32:29.952 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:29 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:29.782+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:29.952 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:29 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:29.782+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:29.952 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:29 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:29.782+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:29.952 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:29 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:29.782+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:29.952 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:29 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:29.782+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:29.952 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 podman[74269]: 2024-03-21 13:32:29.52939307 +0000 UTC m=+0.017680454 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:32:29.953 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 podman[74269]: 2024-03-21 13:32:29.634069334 +0000 UTC m=+0.122356713 container create 20dd0d2e530c4c280980f41817cfa838243b6e20179b3d3dc76bae266597e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-21T13:32:29.953 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 podman[74269]: 2024-03-21 13:32:29.861801268 +0000 UTC m=+0.350088649 container init 20dd0d2e530c4c280980f41817cfa838243b6e20179b3d3dc76bae266597e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True) 2024-03-21T13:32:29.953 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 podman[74269]: 2024-03-21 13:32:29.867525641 +0000 UTC m=+0.355813022 container start 20dd0d2e530c4c280980f41817cfa838243b6e20179b3d3dc76bae266597e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_CLEAN=True) 2024-03-21T13:32:29.953 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:29 smithi152 podman[74269]: 2024-03-21 13:32:29.952805226 +0000 UTC m=+0.441092622 container attach 20dd0d2e530c4c280980f41817cfa838243b6e20179b3d3dc76bae266597e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-21T13:32:30.321 INFO:journalctl@ceph.osd.6.smithi152.stdout:Mar 21 13:32:30 smithi152 podman[74269]: 2024-03-21 13:32:30.157181603 +0000 UTC m=+0.645468976 container died 20dd0d2e530c4c280980f41817cfa838243b6e20179b3d3dc76bae266597e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:32:30.725 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.6.service' 2024-03-21T13:32:30.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:32:30.767 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-21T13:32:30.767 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-21T13:32:30.767 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.7 2024-03-21T13:32:31.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:30 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:30.773+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:31.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:30 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:30.773+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:31.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:30 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:30.773+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:31.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:30 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:30.773+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:31.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:30 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:30.774+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:31.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:30 smithi152 systemd[1]: Stopping Ceph osd.7 for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:31.571 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.097+0000 7fbd90f64700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:32:31.571 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.097+0000 7fbd90f64700 -1 osd.7 787 *** Got signal Terminated *** 2024-03-21T13:32:31.571 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.097+0000 7fbd90f64700 -1 osd.7 787 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:32:32.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.727+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:32.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.727+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:32.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.727+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:32.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.727+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:32.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:31 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:31.727+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:33.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:32 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:32.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:33.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:32 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:32.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:33.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:32 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:32.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:33.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:32 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:32.746+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:33.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:32 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:32.746+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:34.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:33 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:33.740+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:34.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:33 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:33.740+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:34.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:33 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:33.740+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:34.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:33 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:33.740+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:34.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:33 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:33.740+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:34.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:33 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:33.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:34.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:33 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:33.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:34.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:33 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:33.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused" 2024-03-21T13:32:35.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:34 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:34.754+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:35.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:34 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:34.754+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:35.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:34 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:34.754+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:35.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:34 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:34.754+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:35.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:34 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:34.754+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:36.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:35 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:35.715+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-03-21T13:31:38.878236+0000 front 2024-03-21T13:31:38.878428+0000 (oldest deadline 2024-03-21T13:32:02.978196+0000) 2024-03-21T13:32:36.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:35 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:35.715+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6814 osd.1 since back 2024-03-21T13:31:46.880369+0000 front 2024-03-21T13:31:46.880475+0000 (oldest deadline 2024-03-21T13:32:10.980348+0000) 2024-03-21T13:32:36.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:35 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:35.715+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6822 osd.2 since back 2024-03-21T13:31:56.281296+0000 front 2024-03-21T13:31:56.281220+0000 (oldest deadline 2024-03-21T13:32:22.181153+0000) 2024-03-21T13:32:36.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:35 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:35.715+0000 7fbd89550700 -1 osd.7 787 heartbeat_check: no reply from 172.21.15.89:6830 osd.3 since back 2024-03-21T13:32:03.882269+0000 front 2024-03-21T13:32:03.882219+0000 (oldest deadline 2024-03-21T13:32:26.782200+0000) 2024-03-21T13:32:36.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:35 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7[58190]: 2024-03-21T13:32:35.715+0000 7fbd89550700 -1 osd.7 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1227 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e787) 2024-03-21T13:32:36.749 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:36 smithi152 podman[74399]: 2024-03-21 13:32:36.466727321 +0000 UTC m=+5.474209000 container died 68a02c4feb092677797f269f201cc2435023ebc581a78197738628be13c1028b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-21T13:32:37.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:36 smithi152 podman[74399]: 2024-03-21 13:32:36.749786232 +0000 UTC m=+5.757267910 container cleanup 68a02c4feb092677797f269f201cc2435023ebc581a78197738628be13c1028b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.vendor=CentOS) 2024-03-21T13:32:37.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:36 smithi152 bash[74399]: ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7 2024-03-21T13:32:37.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:36 smithi152 podman[74436]: 2024-03-21 13:32:36.992599231 +0000 UTC m=+0.519768973 container remove 68a02c4feb092677797f269f201cc2435023ebc581a78197738628be13c1028b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux ) 2024-03-21T13:32:37.636 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:37 smithi152 podman[74534]: 2024-03-21 13:32:37.227961567 +0000 UTC m=+0.017877317 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:32:37.636 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:37 smithi152 podman[74534]: 2024-03-21 13:32:37.335044042 +0000 UTC m=+0.124959782 container create 2d0dee815dbdf7eece21e9f47c1b5f5a4b5f3fbc777dfb9d4af2e15e8225ed9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T13:32:37.636 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:37 smithi152 podman[74534]: 2024-03-21 13:32:37.54547351 +0000 UTC m=+0.335389252 container init 2d0dee815dbdf7eece21e9f47c1b5f5a4b5f3fbc777dfb9d4af2e15e8225ed9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:32:37.637 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:37 smithi152 podman[74534]: 2024-03-21 13:32:37.550822361 +0000 UTC m=+0.340738102 container start 2d0dee815dbdf7eece21e9f47c1b5f5a4b5f3fbc777dfb9d4af2e15e8225ed9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T13:32:37.637 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:37 smithi152 podman[74534]: 2024-03-21 13:32:37.637008846 +0000 UTC m=+0.426924609 container attach 2d0dee815dbdf7eece21e9f47c1b5f5a4b5f3fbc777dfb9d4af2e15e8225ed9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:32:38.071 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 21 13:32:37 smithi152 podman[74534]: 2024-03-21 13:32:37.841169696 +0000 UTC m=+0.631085440 container died 2d0dee815dbdf7eece21e9f47c1b5f5a4b5f3fbc777dfb9d4af2e15e8225ed9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:32:38.437 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@osd.7.service' 2024-03-21T13:32:38.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:32:38.478 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-21T13:32:38.479 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-21T13:32:38.479 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-21T13:32:38.786 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:38 smithi089 systemd[1]: Stopping Ceph rgw.foo.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:39.074 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a[67512]: 2024-03-21T13:32:38.782+0000 7fcb7f033700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:32:39.074 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:38 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a[67512]: 2024-03-21T13:32:38.782+0000 7fcb8d8b8ac0 -1 shutting down 2024-03-21T13:32:44.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:43 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:43.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:44.075 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:43 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:43.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi089.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.89:8443: connect: connection refused" 2024-03-21T13:32:44.075 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:43 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:43.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi152.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.152:8443: connect: connection refused" 2024-03-21T13:32:49.063 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:48 smithi089 bash[153011]: time="2024-03-21T13:32:48Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-21T13:32:49.063 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:48 smithi089 podman[153011]: 2024-03-21 13:32:48.804092466 +0000 UTC m=+10.132775108 container died d1ba0a72303fe78dfeb01dfa4c72af62c6f56e1f6547e7a9b5176a90db6e9e68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True) 2024-03-21T13:32:49.320 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:49 smithi089 podman[153011]: 2024-03-21 13:32:49.162273429 +0000 UTC m=+10.490956079 container cleanup d1ba0a72303fe78dfeb01dfa4c72af62c6f56e1f6547e7a9b5176a90db6e9e68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T13:32:49.321 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:49 smithi089 bash[153011]: ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-21T13:32:49.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-21T13:32:49.574 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:49 smithi089 podman[153036]: 2024-03-21 13:32:49.321569999 +0000 UTC m=+0.512452637 container remove d1ba0a72303fe78dfeb01dfa4c72af62c6f56e1f6547e7a9b5176a90db6e9e68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-21T13:32:49.574 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:49 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-21T13:32:49.575 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:49 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-21T13:32:49.575 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:49 smithi089 systemd[1]: Stopped Ceph rgw.foo.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:32:49.575 INFO:journalctl@ceph.rgw.foo.a.smithi089.stdout:Mar 21 13:32:49 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@rgw.foo.a.service: Consumed 3.973s CPU time. 2024-03-21T13:32:49.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:32:49.579 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-21T13:32:49.579 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-21T13:32:49.579 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-9c355834-e783-11ee-95cd-87774f69a715@prometheus.a 2024-03-21T13:32:49.902 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 systemd[1]: Stopping Ceph prometheus.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:50.200 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=cleanup t=2024-03-21T13:32:49.919509104Z level=info msg="Completed cleanup jobs" duration=6.341528ms 2024-03-21T13:32:50.200 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:32:50.200 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-21T13:32:50.200 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-21T13:32:50.200 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T13:32:50.200 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a[68978]: ts=2024-03-21T13:32:49.903Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-21T13:32:50.201 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:49 smithi152 podman[74666]: 2024-03-21 13:32:49.913604588 +0000 UTC m=+0.144120439 container died 9a3530fee574c438be3a93fdd0f9b00a85c2486891957f09f261e0ab61ab11a8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:50.533 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:50 smithi152 podman[74666]: 2024-03-21 13:32:50.308090878 +0000 UTC m=+0.538606737 container cleanup 9a3530fee574c438be3a93fdd0f9b00a85c2486891957f09f261e0ab61ab11a8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:50.533 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:50 smithi152 bash[74666]: ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T13:32:50.760 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c355834-e783-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-21T13:32:50.784 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:50 smithi152 podman[74685]: 2024-03-21 13:32:50.534196488 +0000 UTC m=+0.623599723 container remove 9a3530fee574c438be3a93fdd0f9b00a85c2486891957f09f261e0ab61ab11a8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:50.784 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:50 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T13:32:50.784 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:50 smithi152 systemd[1]: Stopped Ceph prometheus.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:32:50.784 INFO:journalctl@ceph.prometheus.a.smithi152.stdout:Mar 21 13:32:50 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 4.393s CPU time. 2024-03-21T13:32:50.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:32:50.797 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-21T13:32:50.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c355834-e783-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T13:32:50.987 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:32:52.803 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:52 smithi089 systemd[1]: Stopping Ceph alertmanager.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:53.074 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:52 smithi089 ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a[78520]: ts=2024-03-21T13:32:52.803Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:32:53.075 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:52 smithi089 podman[153519]: 2024-03-21 13:32:52.844488516 +0000 UTC m=+0.135424841 container died 7c5aa7ad0cbfe98daad9faf14d53a83822789249c013b09c4fe4642f0eaec569 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:53.538 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:53 smithi089 podman[153519]: 2024-03-21 13:32:53.15049159 +0000 UTC m=+0.441427914 container cleanup 7c5aa7ad0cbfe98daad9faf14d53a83822789249c013b09c4fe4642f0eaec569 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:53.538 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:53 smithi089 bash[153519]: ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T13:32:53.538 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:53 smithi089 podman[153539]: 2024-03-21 13:32:53.2847693 +0000 UTC m=+0.435866279 container remove 7c5aa7ad0cbfe98daad9faf14d53a83822789249c013b09c4fe4642f0eaec569 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:53.538 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:53 smithi089 podman[153539]: 2024-03-21 13:32:53.335426795 +0000 UTC m=+0.486523784 volume remove 41c3552979c1db6f21f8f70bbe21c75f467bdb928d708892a09361e8f15236c6 2024-03-21T13:32:53.824 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:53 smithi089 systemd[1]: Stopping Ceph node-exporter.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:53.825 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:53 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T13:32:53.825 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:53 smithi089 systemd[1]: Stopped Ceph alertmanager.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:32:53.825 INFO:journalctl@ceph.alertmanager.a.smithi089.stdout:Mar 21 13:32:53 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.783s CPU time. 2024-03-21T13:32:54.288 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:53 smithi089 podman[153658]: 2024-03-21 13:32:53.993611987 +0000 UTC m=+0.118923643 container died b2da199a1d181e8761848852b23701c697e8f4e08cdca2266b6cbb3146ac5f06 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:54.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:54 smithi089 podman[153658]: 2024-03-21 13:32:54.404631622 +0000 UTC m=+0.529943282 container cleanup b2da199a1d181e8761848852b23701c697e8f4e08cdca2266b6cbb3146ac5f06 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:54.575 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:54 smithi089 bash[153658]: ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T13:32:54.965 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:54 smithi089 podman[153677]: 2024-03-21 13:32:54.589340297 +0000 UTC m=+0.600520158 container remove b2da199a1d181e8761848852b23701c697e8f4e08cdca2266b6cbb3146ac5f06 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:32:54.966 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:54 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T13:32:54.966 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:54 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T13:32:54.966 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:54 smithi089 systemd[1]: Stopped Ceph node-exporter.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:32:54.966 INFO:journalctl@ceph.node-exporter.a.smithi089.stdout:Mar 21 13:32:54 smithi089 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 3.644s CPU time. 2024-03-21T13:32:55.455 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c355834-e783-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T13:32:55.648 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:32:57.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:32:56 smithi152 systemd[1]: Stopping Ceph iscsi.iscsi.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:32:57.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:32:57 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a[63397]: debug Shutdown received 2024-03-21T13:33:07.386 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 bash[75087]: time="2024-03-21T13:33:07Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-21T13:33:07.386 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 podman[75087]: 2024-03-21 13:33:07.133348994 +0000 UTC m=+10.124682554 container died da50c6fe332a93fa2cbf1f71ba542f5a5768e117ab9e39110d96a19597a82802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-21T13:33:07.661 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 podman[75087]: 2024-03-21 13:33:07.444013176 +0000 UTC m=+10.435346737 container cleanup da50c6fe332a93fa2cbf1f71ba542f5a5768e117ab9e39110d96a19597a82802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True) 2024-03-21T13:33:07.661 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 bash[75087]: ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a 2024-03-21T13:33:08.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 podman[75112]: 2024-03-21 13:33:07.661558959 +0000 UTC m=+0.528741967 container remove da50c6fe332a93fa2cbf1f71ba542f5a5768e117ab9e39110d96a19597a82802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-21T13:33:08.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-21T13:33:08.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-21T13:33:08.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 systemd[1]: Stopped Ceph iscsi.iscsi.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:33:08.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi152.stdout:Mar 21 13:33:07 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Consumed 3.453s CPU time. 2024-03-21T13:33:09.738 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:09 smithi152 systemd[1]: Stopping Ceph grafana.a for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:33:10.051 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:09 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=server t=2024-03-21T13:33:09.73813317Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-21T13:33:10.051 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:09 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=ticker t=2024-03-21T13:33:09.738519407Z level=info msg=stopped last_tick=2024-03-21T13:33:00Z 2024-03-21T13:33:10.051 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:09 smithi152 ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a[67480]: logger=tracing t=2024-03-21T13:33:09.738541123Z level=info msg="Closing tracing" 2024-03-21T13:33:10.051 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:09 smithi152 podman[75400]: 2024-03-21 13:33:09.748668172 +0000 UTC m=+0.115703719 container died fe5a4418479c76337ba415619d4cac1d212e2772062b70e020f3af53b4019eec (image=quay.io/ceph/grafana:9.4.12, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:33:10.051 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:10 smithi152 podman[75400]: 2024-03-21 13:33:10.051736912 +0000 UTC m=+0.418772445 container cleanup fe5a4418479c76337ba415619d4cac1d212e2772062b70e020f3af53b4019eec (image=quay.io/ceph/grafana:9.4.12, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:33:10.321 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:10 smithi152 bash[75400]: ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a 2024-03-21T13:33:10.321 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:10 smithi152 podman[75419]: 2024-03-21 13:33:10.244578233 +0000 UTC m=+0.497954594 container remove fe5a4418479c76337ba415619d4cac1d212e2772062b70e020f3af53b4019eec (image=quay.io/ceph/grafana:9.4.12, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:33:10.821 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:10 smithi152 systemd[1]: Stopping Ceph node-exporter.b for 9c355834-e783-11ee-95cd-87774f69a715... 2024-03-21T13:33:10.821 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:10 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-21T13:33:10.822 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:10 smithi152 systemd[1]: Stopped Ceph grafana.a for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:33:10.822 INFO:journalctl@ceph.grafana.a.smithi152.stdout:Mar 21 13:33:10 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@grafana.a.service: Consumed 14.788s CPU time. 2024-03-21T13:33:11.089 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:10 smithi152 podman[75538]: 2024-03-21 13:33:10.82750998 +0000 UTC m=+0.106752365 container died 80fa3287e1972c20909b412d040ed6845b4c10751fb357d3034e62d851b72b3f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:33:11.430 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:11 smithi152 podman[75538]: 2024-03-21 13:33:11.188826406 +0000 UTC m=+0.468068791 container cleanup 80fa3287e1972c20909b412d040ed6845b4c10751fb357d3034e62d851b72b3f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:33:11.431 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:11 smithi152 bash[75538]: ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T13:33:11.711 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:11 smithi152 podman[75557]: 2024-03-21 13:33:11.431489505 +0000 UTC m=+0.608474222 container remove 80fa3287e1972c20909b412d040ed6845b4c10751fb357d3034e62d851b72b3f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9c355834-e783-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:33:11.712 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:11 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T13:33:11.712 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:11 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T13:33:11.712 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:11 smithi152 systemd[1]: Stopped Ceph node-exporter.b for 9c355834-e783-11ee-95cd-87774f69a715. 2024-03-21T13:33:11.712 INFO:journalctl@ceph.node-exporter.b.smithi152.stdout:Mar 21 13:33:11 smithi152 systemd[1]: ceph-9c355834-e783-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 3.654s CPU time. 2024-03-21T13:33:12.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:33:12.217 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:33:12.246 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-21T13:33:12.248 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968/remote/smithi089/crash 2024-03-21T13:33:12.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/crash -- . 2024-03-21T13:33:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:tar: /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T13:33:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:tar: Error is not recoverable: exiting now 2024-03-21T13:33:12.306 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968/remote/smithi152/crash 2024-03-21T13:33:12.307 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/crash -- . 2024-03-21T13:33:12.353 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/9c355834-e783-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T13:33:12.353 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2024-03-21T13:33:12.354 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-21T13:33:12.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-21T13:33:12.388 INFO:teuthology.orchestra.run.smithi089.stdout:2024-03-21T13:14:26.972477+0000 mon.a (mon.0) 898 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-21T13:33:12.388 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-21T13:33:12.389 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[SEC\]' /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-21T13:33:12.460 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]' /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-21T13:33:12.531 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[WRN\]' /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-21T13:33:12.601 INFO:teuthology.orchestra.run.smithi089.stdout:2024-03-21T13:14:26.972477+0000 mon.a (mon.0) 898 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-21T13:33:12.602 INFO:tasks.cephadm:Compressing logs... 2024-03-21T13:33:12.602 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T13:33:12.644 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T13:33:12.668 INFO:teuthology.orchestra.run.smithi089.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T13:33:12.668 INFO:teuthology.orchestra.run.smithi089.stderr:: No such file or directory 2024-03-21T13:33:12.668 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-21T13:33:12.668 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T13:33:12.671 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-21T13:33:12.671 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-21T13:33:12.671 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-21T13:33:12.671 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-21T13:33:12.671 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-21T13:33:12.671 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T13:33:12.673 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T13:33:12.673 INFO:teuthology.orchestra.run.smithi152.stderr:: No such file or directory 2024-03-21T13:33:12.673 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-21T13:33:12.673 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T13:33:12.673 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-21T13:33:12.673 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.6.log: /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-21T13:33:12.673 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-21T13:33:12.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.4.log: /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-21T13:33:12.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-21T13:33:12.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T13:33:12.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/tcmu-runner.log: 68.0% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-21T13:33:12.678 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log 2024-03-21T13:33:12.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T13:33:12.678 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T13:33:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.0.log: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T13:33:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log 2024-03-21T13:33:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.cephadm.log: 89.1% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T13:33:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-21T13:33:12.683 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log: 87.1% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T13:33:12.683 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-21T13:33:12.683 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-volume.log: 92.5% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T13:33:12.684 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.b.log: 92.1% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-21T13:33:12.685 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log: 92.8% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T13:33:12.685 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T13:33:12.686 INFO:teuthology.orchestra.run.smithi152.stderr: 91.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T13:33:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.2.log: 95.3% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T13:33:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-21T13:33:12.695 INFO:teuthology.orchestra.run.smithi152.stderr: 91.0% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T13:33:12.702 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T13:33:12.717 INFO:teuthology.orchestra.run.smithi089.stderr: 94.1% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-21T13:33:13.108 INFO:teuthology.orchestra.run.smithi089.stderr: 91.2% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-21T13:33:13.701 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-21T13:33:14.084 INFO:teuthology.orchestra.run.smithi089.stderr: 92.5% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-21T13:33:14.493 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-21T13:33:19.076 INFO:teuthology.orchestra.run.smithi152.stderr: 93.8% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-21T13:33:19.476 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-21T13:33:19.636 INFO:teuthology.orchestra.run.smithi152.stderr: 93.9% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-21T13:33:19.696 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-21T13:33:19.808 INFO:teuthology.orchestra.run.smithi152.stderr: 93.8% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-21T13:33:19.881 INFO:teuthology.orchestra.run.smithi152.stderr: 93.8% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-21T13:33:19.883 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-21T13:33:19.883 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m7.225s 2024-03-21T13:33:19.883 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m27.752s 2024-03-21T13:33:19.884 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m1.013s 2024-03-21T13:33:20.044 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-21T13:33:20.084 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/9c355834-e783-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-21T13:33:20.086 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-21T13:33:20.086 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m7.430s 2024-03-21T13:33:20.086 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m31.180s 2024-03-21T13:33:20.086 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m1.111s 2024-03-21T13:33:20.086 INFO:tasks.cephadm:Archiving logs... 2024-03-21T13:33:20.087 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968/remote/smithi089/log 2024-03-21T13:33:20.088 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T13:33:24.296 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968/remote/smithi152/log 2024-03-21T13:33:24.297 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T13:33:28.343 INFO:tasks.cephadm:Removing cluster... 2024-03-21T13:33:28.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c355834-e783-11ee-95cd-87774f69a715 --force 2024-03-21T13:33:28.534 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:33:28.827 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c355834-e783-11ee-95cd-87774f69a715 --force 2024-03-21T13:33:29.020 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 9c355834-e783-11ee-95cd-87774f69a715 2024-03-21T13:33:29.309 INFO:tasks.cephadm:Removing cephadm ... 2024-03-21T13:33:29.309 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T13:33:29.325 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T13:33:29.341 INFO:tasks.cephadm:Teardown complete 2024-03-21T13:33:29.341 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-21T13:33:29.369 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-21T13:33:29.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T13:33:29.372 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T13:33:29.457 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-21T13:33:29.471 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-21T13:33:29.471 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:33:29.473 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:33:29.485 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:33:29.487 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:33:29.493 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:33:29.493 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-03-21T13:33:29.493 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +2299us[+2286us] +/- 111ms 2024-03-21T13:33:29.493 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -1307us[-1320us] +/- 51ms 2024-03-21T13:33:29.494 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +212us[ +212us] +/- 66ms 2024-03-21T13:33:29.494 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:33:29.496 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:33:29.496 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-21T13:33:29.496 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +2290us[+2290us] +/- 111ms 2024-03-21T13:33:29.497 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -1305us[-1304us] +/- 51ms 2024-03-21T13:33:29.497 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +228us[ +228us] +/- 66ms 2024-03-21T13:33:29.497 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:33:29.497 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-21T13:33:29.509 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-21T13:33:29.509 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-21T13:33:29.519 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T13:33:29.612 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T13:33:29.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T13:33:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:33:29.718 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T13:33:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:33:29.749 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-21T13:33:29.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-21T13:33:29.769 INFO:teuthology.task.internal:Duration was 2130.649338 seconds 2024-03-21T13:33:29.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-21T13:33:29.781 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-21T13:33:29.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T13:33:29.784 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T13:33:29.824 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:33:29.831 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:33:30.107 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-21T13:33:30.107 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-03-21T13:33:30.107 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T13:33:30.166 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-03-21T13:33:30.166 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T13:33:30.191 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-21T13:33:30.191 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T13:33:30.208 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T13:33:30.261 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-21T13:33:30.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T13:33:30.276 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T13:33:30.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-21T13:33:30.708 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-21T13:33:30.709 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T13:33:30.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T13:33:30.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-21T13:33:30.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:33:30.778 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:33:30.801 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-03-21T13:33:30.828 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-03-21T13:33:30.876 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:33:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:33:30.890 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:33:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:33:30.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-21T13:33:30.942 INFO:teuthology.task.internal:Transferring archived files... 2024-03-21T13:33:30.943 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968/remote/smithi089 2024-03-21T13:33:30.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T13:33:31.006 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968/remote/smithi152 2024-03-21T13:33:31.006 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T13:33:31.045 INFO:teuthology.task.internal:Removing archive directory... 2024-03-21T13:33:31.045 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T13:33:31.048 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T13:33:31.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-21T13:33:31.108 INFO:teuthology.task.internal:Not uploading archives. 2024-03-21T13:33:31.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-21T13:33:31.123 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-21T13:33:31.123 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T13:33:31.125 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T13:33:31.137 INFO:teuthology.orchestra.run.smithi089.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 13:33 /home/ubuntu/cephtest 2024-03-21T13:33:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: 265575 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 13:33 /home/ubuntu/cephtest 2024-03-21T13:33:31.149 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-21T13:33:31.161 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-21T13:33:31.192 INFO:teuthology.nuke:Checking targets against current locks 2024-03-21T13:33:31.217 DEBUG:teuthology.nuke:shortname: smithi089 2024-03-21T13:33:31.218 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T13:33:31.244 DEBUG:teuthology.nuke:shortname: smithi152 2024-03-21T13:33:31.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T13:33:31.248 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:48:52.449483', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDW2mtFwLHDI53XM5x30fN+QMr0GjYn8Z9SGaI2yhRbTGeZ/a6yDZeKmK9DTnHfqDfbtY6MycCsg8wvR/boRIBQ='} 2024-03-21T13:33:31.274 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614968', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:48:52.451928', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJVRSq3H3HXMpH8erFtBRpPhkjH2VgB31174t5l1L/huEQKypktSEaC/3a2bR1kYBdKOpizdGSX9sRSbgXBDqE='} 2024-03-21T13:33:31.301 INFO:teuthology.orchestra.console.smithi089:Power off 2024-03-21T13:33:31.301 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T13:33:31.325 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-03-21T13:33:31.325 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:33:31.341 INFO:teuthology.orchestra.console.smithi152:Power off 2024-03-21T13:33:31.342 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T13:33:31.350 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-21T13:33:31.364 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-03-21T13:33:31.364 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:33:31.384 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-21T13:33:35.351 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:33:35.386 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:33:35.461 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-21T13:33:35.494 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-21T13:33:39.466 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:33:39.496 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:33:39.576 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-03-21T13:33:39.605 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-21T13:33:39.677 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-03-21T13:33:43.609 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:33:43.720 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-03-21T13:33:43.820 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-03-21T13:33:43.924 INFO:teuthology.run:Summary data: description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} duration: 2130.649337530136 failure_reason: '"2024-03-21T13:14:26.972477+0000 mon.a (mon.0) 898 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log' flavor: default owner: scheduled_gabrioux@teuthology success: false 2024-03-21T13:33:43.924 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T13:33:44.002 INFO:teuthology.run:FAIL